Discussion regarding an implementation of JSDL
All, I represent the Thebes middleware consortium, which is hosted at Georgetown University. One of the things the Thebes consortium is working on is a job acceptance service that can accept jobs, translate them to DRMAA, and submit them to job schedulers. Our intention is to include the ability to accept SAML assertions to make policy decisions. We currently use an axis web service that takes JSDL and uses DRMAA to submit the job to Sun Grid Engine. It is rudimentary at this point. We're interested in projects that are working in similar arenas, and we're interested in thoughts from the JSDL team about recommendations, suggestions, etc. Thanks, Arnie Miles Middleware Architect Georgetown University
Hi Arnie, On Fri, Jan 16, 2009 at 6:55 PM, Arnie Miles <adm35@georgetown.edu> wrote:
I represent the Thebes middleware consortium, which is hosted at Georgetown University.
One of the things the Thebes consortium is working on is a job acceptance service that can accept jobs, translate them to DRMAA, and submit them to job schedulers. Our intention is to include the ability to accept SAML assertions to make policy decisions.
We currently use an axis web service that takes JSDL and uses DRMAA to submit the job to Sun Grid Engine. It is rudimentary at this point.
We're interested in projects that are working in similar arenas, and we're interested in thoughts from the JSDL team about recommendations, suggestions, etc.
PSNC (Poznan Supercomputing and Networking Center) has been working on the exact same piece of software, called SMOA Computing, for some time now. I've been involved in this project for some time. Major features include: - C implementation using gSOAP, it uses SMOA Core, our library with standard interface for service configuration, loadable modules, and other neat stuff. - SMOA Core has modules for various authentication mechanisms and authorization policies. It has an ability to accept SAML assertions as an authentication mechanism. Besides, you may currently use plain HTTP, SSL (with client authentication), GSI or WS-Security Username. - DRMAA interface to job schedulers (we mostly use it with SGE and LSF). Remote users are mapped to local uids. - Web Service interface compliant with OGSA Basic Execution Service Version 1.0 enchanced with our extensions. - JSDL HPC Profile Application Extension, Version 1.0. - HPC File Staging Profile, Version 1.0. - HPC Basic Profile, Version 1.0. - It's customizable with easy to write Python or C plugins. For example, one can write a plugin that reacts to job state changes (e.g. sends out a notification). - Includes Advance Reservation port that allows you to reserve resources in advance and then submit jobs within these reservations. Supported with SGE and LSF. - Work is currently under way to implement JSDL Parameter Sweep. Cheers, -- Piotr Domagalski
Thank you Piotr, A couple of questions embedded;
- SMOA Core has modules for various authentication mechanisms and authorization policies. It has an ability to accept SAML assertions as an authentication mechanism. Besides, you may currently use plain HTTP, SSL (with client authentication), GSI or WS-Security Username.
Do you have any installations that use SAML? What is creating the assertions in these installations?
- DRMAA interface to job schedulers (we mostly use it with SGE and LSF). Remote users are mapped to local uids.
Is this mapping of users to local uids done "on the fly" or in advance? What mechanisms are you using for tracking accounting statistics and enforcing policies? Thanks, Arnie
Hi Arnie, On Mon, Jan 19, 2009 at 2:53 PM, Arnie Miles <adm35@georgetown.edu> wrote:
Thank you Piotr,
A couple of questions embedded;
- SMOA Core has modules for various authentication mechanisms and authorization policies. It has an ability to accept SAML assertions as an authentication mechanism. Besides, you may currently use plain HTTP, SSL (with client authentication), GSI or WS-Security Username.
Do you have any installations that use SAML? What is creating the assertions in these installations?
- DRMAA interface to job schedulers (we mostly use it with SGE and LSF). Remote users are mapped to local uids.
Is this mapping of users to local uids done "on the fly" or in advance? What mechanisms are you using for tracking accounting statistics and enforcing policies?
some time ago we have successfully realized following scenario: the client authenticating to SMOA Computing using SAML bearer assertion. The assertion was issued by the other entity - Liberty ID-WSF Single Sign On Service (acting as the Security Token Service - STS). The client authenticated to the STS using simply username and password. We are working on supporting any mechanism that can be expressed using SASL message pattern (Liberty Authentication Service and Discovery Service). there are many options: - while not using any authentication mechanism each user can be mapped to one fixed local uid or to the user specified in JSDL document. - while using some transport/message level security mechanism (SSL, GSI, SAML, Username Token) the local user is determined either statically (using local mapfile) or dynamically (by issuing call to external Grid Authorization Service) based on provided security context (X.509 DN, SAML Subject name) - all information about jobs (JSDL, start/finish time, resources usage information) are stored into database.
Thanks, Arnie
Cheers, -- Mariusz
One of the things the Thebes consortium is working on is a job acceptance service that can accept jobs, translate them to DRMAA, and submit them to job schedulers. Our intention is to include the ability to accept SAML assertions to make policy decisions.
The Basic Execution Service and the HPC Basic Profile are well established web service interfaces that take in a web service message. Their use with other specifications to run jobs on remote computing resources is described in the ISV Remote Computing primer (http://www.ogf.org/gf/docs/gfd.php?gfd=141) with implementations by open source and commercial software providers. I would suggest looking at these. The use of DRMAA within the web service to submit to a local scheduler is great to get portability of the web service to different schedulers. If you have requirements beyond this you might want to look at the work that is just starting in the OGF in the Production Grid Infrastructure Working Group - http://forge.gridforum.org/sf/projects/pgi-wg. Details of all the specifications can be found through the OGF web pages. Steven
Steven, Thank you for your reply. I'll study your reference. In the meanwhile, could you give me pointers to the implementations you've referred to? Thanks! Arnie Steven Newhouse wrote:
One of the things the Thebes consortium is working on is a job acceptance service that can accept jobs, translate them to DRMAA, and submit them to job schedulers. Our intention is to include the ability to accept SAML assertions to make policy decisions.
The Basic Execution Service and the HPC Basic Profile are well established web service interfaces that take in a web service message. Their use with other specifications to run jobs on remote computing resources is described in the ISV Remote Computing primer (http://www.ogf.org/gf/docs/gfd.php?gfd=141) with implementations by open source and commercial software providers. I would suggest looking at these.
The use of DRMAA within the web service to submit to a local scheduler is great to get portability of the web service to different schedulers.
If you have requirements beyond this you might want to look at the work that is just starting in the OGF in the Production Grid Infrastructure Working Group - http://forge.gridforum.org/sf/projects/pgi-wg. Details of all the specifications can be found through the OGF web pages.
Steven
Arnie, There are various open source implementations of the HPC Basic Profile. Look at the work of Andrew Grimshaw from the University of Virginia (Genesis II) and the SourceForge project (BES++) build by Chris Smith from Platform. There are closed source implementations in the Microsoft HPC product. There are others... I've cc'ed these two people if you have difficulties in accessing these projects. Thanks, Steven Dr Steven Newhouse EGEE Technical Director http://cern.ch/Steven.Newhouse
-----Original Message----- From: Arnie Miles [mailto:adm35@georgetown.edu] Sent: 19 January 2009 14:45 To: Steven Newhouse Cc: jsdl-wg@ogf.org Subject: Re: [jsdl-wg] Discussion regarding an implementation of JSDL
Steven,
Thank you for your reply. I'll study your reference. In the meanwhile, could you give me pointers to the implementations you've referred to?
Thanks! Arnie
Steven Newhouse wrote:
One of the things the Thebes consortium is working on is a job acceptance service that can accept jobs, translate them to DRMAA, and submit them to job schedulers. Our intention is to include the ability to accept SAML assertions to make policy decisions.
The Basic Execution Service and the HPC Basic Profile are well established web service interfaces that take in a web service message. Their use with other specifications to run jobs on remote computing resources is described in the ISV Remote Computing primer (http://www.ogf.org/gf/docs/gfd.php?gfd=141) with implementations by open source and commercial software providers. I would suggest looking at these.
The use of DRMAA within the web service to submit to a local scheduler is great to get portability of the web service to different schedulers.
If you have requirements beyond this you might want to look at the work that is just starting in the OGF in the Production Grid Infrastructure Working Group - http://forge.gridforum.org/sf/projects/pgi-wg. Details of all the specifications can be found through the OGF web pages.
Steven
Arnie, There is some overlap with documents already mentioned but you could also have a look at the JSDL Experience document (http://www.ogf.org/documents/GFD.140.pdf). There is some information on what some projects did or planned to do around security. I found two mentions of saml though none of drmaa in the document. On Mon, 19 Jan 2009 23:10:21 +0900, Steven Newhouse <Steven.Newhouse@cern.ch> wrote:
Arnie,
There are various open source implementations of the HPC Basic Profile. Look at the work of Andrew Grimshaw from the University of Virginia (Genesis II) and the SourceForge project (BES++) build by Chris Smith from Platform. There are closed source implementations in the Microsoft HPC product.
There are others... I've cc'ed these two people if you have difficulties in accessing these projects.
Thanks,
Steven
Dr Steven Newhouse EGEE Technical Director http://cern.ch/Steven.Newhouse
-- Andreas Savva Fujitsu Laboratories Ltd
Sorry about the delayed response, I've been out. I want to thank everyone who responded. Now I'm going to review everything that I received. I will also start publishing our work for others to look at as soon as we have something worth sharing. Thanks again, Arnie Andreas Savva wrote:
Arnie,
There is some overlap with documents already mentioned but you could also have a look at the JSDL Experience document (http://www.ogf.org/documents/GFD.140.pdf). There is some information on what some projects did or planned to do around security. I found two mentions of saml though none of drmaa in the document.
On Mon, 19 Jan 2009 23:10:21 +0900, Steven Newhouse <Steven.Newhouse@cern.ch> wrote:
Arnie,
There are various open source implementations of the HPC Basic Profile. Look at the work of Andrew Grimshaw from the University of Virginia (Genesis II) and the SourceForge project (BES++) build by Chris Smith from Platform. There are closed source implementations in the Microsoft HPC product.
There are others... I've cc'ed these two people if you have difficulties in accessing these projects.
Thanks,
Steven
Dr Steven Newhouse EGEE Technical Director http://cern.ch/Steven.Newhouse
Hello Arnie,
Thank you for your reply. I'll study your reference. In the meanwhile, could you give me pointers to the implementations you've referred to?
One of those solutions that implement BES and JSDL (also mentioned in that document) is ARC, see e.g. http://svn.nordugrid.org/repos/nordugrid/arc1/trunk/ There's no DRMAA in ARC though. Cheers, Oxana
Thanks! Arnie
Steven Newhouse wrote:
One of the things the Thebes consortium is working on is a job acceptance service that can accept jobs, translate them to DRMAA, and submit them to job schedulers. Our intention is to include the ability to accept SAML assertions to make policy decisions.
The Basic Execution Service and the HPC Basic Profile are well established web service interfaces that take in a web service message. Their use with other specifications to run jobs on remote computing resources is described in the ISV Remote Computing primer (http://www.ogf.org/gf/docs/gfd.php?gfd=141) with implementations by open source and commercial software providers. I would suggest looking at these.
The use of DRMAA within the web service to submit to a local scheduler is great to get portability of the web service to different schedulers.
If you have requirements beyond this you might want to look at the work that is just starting in the OGF in the Production Grid Infrastructure Working Group - http://forge.gridforum.org/sf/projects/pgi-wg. Details of all the specifications can be found through the OGF web pages.
Steven
-- jsdl-wg mailing list jsdl-wg@ogf.org http://www.ogf.org/mailman/listinfo/jsdl-wg
participants (6)
-
Andreas Savva -
Arnie Miles -
Mariusz Mamoński -
Oxana Smirnova -
Piotr Domagalski -
Steven Newhouse