Dear all, we discussed a list of open issues for the DRMAAv2 root specification and C binding. Here is a summary of the decisions: - Root specification and C binding get a new function „reap()“ in the Job interface: http://redmine.ogf.org/issues/163 - The C binding gets a new function „exit()“ on global scope: http://redmine.ogf.org/issues/103 - Root specification and C binding get a new attribute „jobName“ in the „JobInfo“ structure: http://redmine.ogf.org/issues/102 - The C binding will define the file name for the implementation-specific header file: http://redmine.ogf.org/issues/162 - All extensible structures in the C binding get an additional attribute, pointing to the impl.-specific data structure: http://redmine.ogf.org/issues/160 - The C binding gets methods for list copying and dictionary copying: http://redmine.ogf.org/issues/58 - C binding constants shall be declared as extern, so that the implementation defines them: http://redmine.ogf.org/issues/57 - Minor text clarifications and method name fixes in root specification and C binding: http://redmine.ogf.org/issues/61 http://redmine.ogf.org/issues/114 http://redmine.ogf.org/issues/113 http://redmine.ogf.org/issues/104 http://redmine.ogf.org/issues/59 http://redmine.ogf.org/issues/165 For the following issue, we came to no feasible conclusion, so your feedback is desperately demanded: http://redmine.ogf.org/issues/116 http://redmine.ogf.org/issues/115 Please note that DRMAAv2 (C binding) is now shipped with Univa Grid Engine 6.2 as official part of the product, which is a huge step forward. This also makes the development and testing of language bindings much easier. Daniel Gruber gave us some insight into the Go language binding and the according prototype implementation: http://redmine.ogf.org/projects/drmaav2-go-binding/repository/revisions/mast... https://github.com/dgruber/drmaa/tree/master Due to several requests, we are now also actively targeting the Python and Java language binding. Check for updates in the OGF Redmine pages. Best regards, Peter.
Hi Peter! Unfortunately I'm new to DRMAA and can't contribute much to the decision, but thanks a lot for summarizing it. I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors. Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc. Thanks Bruno
________________________________ From: Peter Tröger <peter@troeger.eu> To: drmaa-wg <drmaa-wg@ogf.org> Sent: Tuesday, September 9, 2014 8:06 PM Subject: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Dear all,
we discussed a list of open issues for the DRMAAv2 root specification and C binding. Here is a summary of the decisions:
- Root specification and C binding get a new function „reap()“ in the Job interface: http://redmine.ogf.org/issues/163 - The C binding gets a new function „exit()“ on global scope: http://redmine.ogf.org/issues/103 - Root specification and C binding get a new attribute „jobName“ in the „JobInfo“ structure: http://redmine.ogf.org/issues/102 - The C binding will define the file name for the implementation-specific header file: http://redmine.ogf.org/issues/162 - All extensible structures in the C binding get an additional attribute, pointing to the impl.-specific data structure: http://redmine.ogf.org/issues/160 - The C binding gets methods for list copying and dictionary copying: http://redmine.ogf.org/issues/58 - C binding constants shall be declared as extern, so that the implementation defines them: http://redmine.ogf.org/issues/57 - Minor text clarifications and method name fixes in root specification and C binding: http://redmine.ogf.org/issues/61 http://redmine.ogf.org/issues/114 http://redmine.ogf.org/issues/113 http://redmine.ogf.org/issues/104 http://redmine.ogf.org/issues/59 http://redmine.ogf.org/issues/165
For the following issue, we came to no feasible conclusion, so your feedback is desperately demanded: http://redmine.ogf.org/issues/116 http://redmine.ogf.org/issues/115
Please note that DRMAAv2 (C binding) is now shipped with Univa Grid Engine 6.2 as official part of the product, which is a huge step forward. This also makes the development and testing of language bindings much easier.
Daniel Gruber gave us some insight into the Go language binding and the according prototype implementation:
http://redmine.ogf.org/projects/drmaav2-go-binding/repository/revisions/mast... https://github.com/dgruber/drmaa/tree/master
Due to several requests, we are now also actively targeting the Python and Java language binding. Check for updates in the OGF Redmine pages.
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
Dear Bruno, thanks for your feedback, we always need encouraged people to join the DRMAA working group.
I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.
Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step. Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1). Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways. Best regards, Peter.
Hello Peter
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.
That's our plan. And we also believe that using an industry-proven software, with formed community and documentation will make it simpler for contributors and new developers.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
That sounds great. I'll try to hack an ansible + vagrant/docker script to give it a shot.
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
Thanks! Are you going to use GitHub? If so I can probably bother you there with issues and maybe PR's :) All the best, Bruno
________________________________ From: Peter Tröger <peter@troeger.eu> To: Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> Cc: drmaa-wg <drmaa-wg@ogf.org> Sent: Wednesday, September 10, 2014 4:14 AM Subject: Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Dear Bruno,
thanks for your feedback, we always need encouraged people to join the DRMAA working group.
I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.
Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
Best regards, Peter.
Hi Bruno, I've seen your blog posts on possibly using SWIG or JNI for DRMAA. Back with DRMAAv1, with both a C <http://redmine.ogf.org/documents/10> and a Java <http://redmine.ogf.org/documents/11> specs available, we implemented a generic DRMAAv1 Java-to-C binding using JNA <https://github.com/twall/jna>'s dynamic invocation. While I never had time to support our wrapper <https://github.com/broadgsa/gatk/tree/9d15d241af5d4a3709e3c76249cff4c2a9053d60/public/gatk-tools-public/src/main/java/org/broadinstitute/gatk/utils/jna/drmaa/v1_0> as a proper mavenized library, I'd love to see someone publish a similar DRMAAv2 JNA wrapper as an artifact, of course after both specs are out. Anecdotally, after a user contributed patch mapped other parts of our software to PBS specific variables, the above wrapper seemed to work for at least a another <http://gatkforums.broadinstitute.org/discussion/2150/queue-and-pbs> without further modification to our DRMAAv1 JNA code. While I haven't measured the actual speed of GridEngine-via-JNA vs. GridEngine-via-JNI, this dynamic wrapper "feels" fast enough for most users, as the JNA probably adds <0.5s per call, ever. This known cost <http://developers.opengamma.com/blog/2012/05/25/jna-jni-and-raw-java-performance> has been worth it since we now only include a single DRMAAv1-JNA wrapper, versus multiple JNI wrappers for DRMAAv1-PBS, DRMAAv1-GridEngine, etc. Best wishes, -k On Wed, Sep 10, 2014 at 9:10 PM, Bruno P. Kinoshita < brunodepaulak@yahoo.com.br> wrote:
Hello Peter
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.
That's our plan. And we also believe that using an industry-proven software, with formed community and documentation will make it simpler for contributors and new developers.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
That sounds great. I'll try to hack an ansible + vagrant/docker script to give it a shot.
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
Thanks! Are you going to use GitHub? If so I can probably bother you there with issues and maybe PR's :)
All the best, Bruno
------------------------------ *From:* Peter Tröger <peter@troeger.eu> *To:* Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> *Cc:* drmaa-wg <drmaa-wg@ogf.org> *Sent:* Wednesday, September 10, 2014 4:14 AM *Subject:* Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Dear Bruno,
thanks for your feedback, we always need encouraged people to join the DRMAA working group.
I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.
Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
Hello Khalid Very interesting links, thanks! I haven't used JNI or JNA for quite some time. Supporting Maven is a requirement for us, so if the DRMAA v2 isn't released to Maven Central we will probably fork and release it under a different groupId. Due to Jenkins master/slave architecture, we have two models in mind. One where the user will start a slave remotely in one computer with access to submit jobs and manage the queue (here we can use JNA or JNI). And the other model, where installing a slave (some Java code communicating with the master via SSH) won't be necessary, and we will use overthere [1] to send commands and transfer files via SSH. Our main requirements in this Jenkins plug-in are to submit Jobs but be able to mix local processing and cluster processing, and manage the queue remotely, so that researchers won't have to log in to check the status of the submitted jobs. In any case, we will probably have to use the DRMAA v2 Java binding, but for the latter case write custom remote Java APIs (one for PBS, one for SGE, and so it goes) that don't use JNI or JNA, but sends commands via SSH and parses the output. It is not clear which is the best way, so we will work with the researchers interested in testing the plug-in to see which model works best. Thanks! Bruno [1] https://github.com/xebialabs/overthere
________________________________ From: Khalid Shakir <kshakir@broadinstitute.org> To: Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> Cc: drmaa-wg <drmaa-wg@ogf.org> Sent: Wednesday, September 10, 2014 3:15 PM Subject: Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Hi Bruno,
I've seen your blog posts on possibly using SWIG or JNI for DRMAA. Back with DRMAAv1, with both a C and a Java specs available, we implemented a generic DRMAAv1 Java-to-C binding using JNA's dynamic invocation. While I never had time to support our wrapper as a proper mavenized library, I'd love to see someone publish a similar DRMAAv2 JNA wrapper as an artifact, of course after both specs are out.
Anecdotally, after a user contributed patch mapped other parts of our software to PBS specific variables, the above wrapper seemed to work for at least a another without further modification to our DRMAAv1 JNA code. While I haven't measured the actual speed of GridEngine-via-JNA vs. GridEngine-via-JNI, this dynamic wrapper "feels" fast enough for most users, as the JNA probably adds <0.5s per call, ever. This known cost has been worth it since we now only include a single DRMAAv1-JNA wrapper, versus multiple JNI wrappers for DRMAAv1-PBS, DRMAAv1-GridEngine, etc.
Best wishes, -k
On Wed, Sep 10, 2014 at 9:10 PM, Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> wrote:
Hello Peter
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.
That's our plan. And we also believe that using an industry-proven software, with formed community and documentation will make it simpler for contributors and new developers.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
That sounds great. I'll try to hack an ansible + vagrant/docker script to give it a shot.
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
Thanks! Are you going to use GitHub? If so I can probably bother you there with issues and maybe PR's :)
All the best, Bruno
________________________________ From: Peter Tröger <peter@troeger.eu> To: Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> Cc: drmaa-wg <drmaa-wg@ogf.org> Sent: Wednesday, September 10, 2014 4:14 AM Subject: Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Dear Bruno,
thanks for your feedback, we always need encouraged people to join the DRMAA working group.
I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.
Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
participants (3)
-
Bruno P. Kinoshita -
Khalid Shakir -
Peter Tröger