Thoughts on the airplane home (GIN, PGI, Cycle sharing)

All, I thought we had a very productive OGF. A number of interesting and exciting results in GIN, and solid progress on PGI requirements and a discussion of how to meet those requirements going forward. A couple of things stuck in my mind - so I thought I'd get them out there and get a response. First, with respect to GIN. I thought the demo's, particularly the use of the BES endpoints in SAGA, were very informative. Particularly when one keeps in mind that at least the Unicore/futuregrid resources were not even available until Friday afternoon (eastern time!). I agree with the sentiment that we should build on this very successful demo. In particular most groups have either done simple extensions to JSDL, or used existing mechanism in a particular way. Ditto with BES factory attributes. Several ideas came up (Morris, chime in I think you were keeping notes) Profiling we can do fairly easily: File system JSDL specifies how file systems can be defined - Mark correct me on this. We use one we call SCRATCH as a place to cache files that are large and constant, including often zip files that contain applications. We tell JSDL to mount it, we then use the "don't overwrite on stage-in" JSDL flag, and the "don't delete when you're done" flag. This is a huge help. Extra attributes We have found a need for several extra BES factory attributes and the use of those in JSDL matching. I'm guessing many of you have done this as well. These include whether an application is installed, whether static linking is required, I also think that while we are waiting for PGI to figure out how to deploy applications that we have a simple convention for applications, something like HAS_APPX_INSTALLED And then if the BES factory has that, then by convention that APPX_PATH and APPX_LIB_PATH are defined and can be used in some sort of prefix. File staging - demo of sites that support FSE at SC. We discussed that several implementations support FSE. Let's test that interop. It really changes the set of applications you can run if you can copy them onto the machine. If we can get a set of endpoints that support FSE we'll (GenesisII/XCG) set up a meta-scheduler that takes JSDL and manage them through the life cycle on the endpoints of the different implementations .This might also form the basis for some cycle trading ... Re: FutureGrid. I also want to repeat that if groups want to put up standards based implementations of services on FutureGrid (OGSA-BES, RNS, WS-Trust, ByteIO, gridFTP, SRM) we will work with them to make it so. One of the goals of FutureGrid is to provide PERSISTENT standards based endpoints with which to test. Second, with respect to PGI Application deployment/management. This came across as the number one requirement. This came as a surprise to me, though I agree it is an important next step to make execution services (OGSA-BES) more useful. There are at least two ways to handle this problem - one I'll call manual installation, configuration, and reification (MICR), and the other I'll call automatic deployment, configuration, and reification (ADCR). MICR This is definitely the easy case. First, an application is deployed at/on a site/compute resource. Appropriate metadata is added to the resource metadata or an information service is updated with information on the local bindings for the executable, paths, libraries, etc. Second, once a placement decision is made by a job manager (I'm using the OGSA-EMS term, here think meta-scheduler), the JSDL document that contains an abstract application name is reified (transformed) to have the site-appropriate values for the application. Third, the JSDL is sent to the compute resource. I believe this is basically how UNICORE 6 does this now. ADCR ADCR is similar to MICR except the deploying step of setting the application up and configuring the metadata is done automatically by a service. For simple applications - a single executable, or maybe a zip file that needs to be unzipped, this is easy. It can get complicated really fast. We at Genesis II defined a very simple deploying service combined with a reification service that worked for the easy case - executable and zip file. I can dig up the documentation on this if anyone is interested. I believe that the general case for this is a tarpit for a standards body. I have a PhD student working on it J. **** A note on reification
From Wikipedia
Reification generally refers to bringing into being or turning concrete. Specifically, reification may refer to: * Reification (computer science) <http://en.wikipedia.org/wiki/Reification_%28computer_science%29> , making a data model for a previously abstract concept In this context I refer to taking an abstract JSDL that may give an abstract name of an application, e.g., <JSDL-abstract-application-name>/bin/biology/blastp-v5 </ JSDL-abstract-application-name> THIS IS MADE UP SYNTAX That might occur in a JSDL file, and then making it concrete for a particular endpoint by transforming the JSDL to have the actual local path name and defining any environment variables needed. We discussed this problem extensively in the OGSA, OGSA-EMS, and OGSA-BES working groups. A strawman interface was even defined. MPI Once again the importance of this came as a somewhat welcome surprise. I believe this is of considerable interest to UNICORE. Perhaps they could take the lead on this one. Is it simply a problem of the JSDL not being expressive enough to both select appropriate resources and pass topology information down to MPI? In other words, is this an easy problem? Or is there something else I am not seeing? Security The single greatest impediment to the GIN demonstrations was working out the security stuff. Andre was pretty clear about this. In other words, the functional aspects of the standards were not the problem - the certificates and other stuff were. It has been suggested that it may be time to dive back into this. We spent quite a bit of time on it 18 months ago. The result was a DRAFT profile document by Duane Merrill that bifurcated the profile into a transport level security profile based on delegated X.509 certificates and a message level security profile - primarily aimed at SAML (http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi-wg/do cman.root.input_documents.security_material.comm_profile/doc15575) . I know that there was a lively debate about how to proceed. PGI profile This is the big question. We've gone through use cases, requirements, counting up the use cases that require each requirement, and two approaches to meeting those requirements (see slides). How are we to proceed? Should we ask each proposer to provide more detail? Should we solicit more proposals? Should we work through both paths? Should we vote? My own interest is in extending the existing specs and working with GIN to drive spec modifications. If we go with extending the existing specs - BES in particular, should we modify BES just a little and leave the basic factory and management pattern the same? Or go with a cleaner model more consistent with what Bernd presented (return an activity handle <EPR?> and then manage the activity directly). Either way we need make some decisions and move forward. Third, with respect to Cycle sharing At first I thought that the cycle sharing BOF did not lead to any interested in trying this out. As the week wore on my initial impression proved mistaken. Johannes followed up with several other groups and we may indeed have a sufficient critical mass to proceed. I will ask Joel to set up a source forge site and email list. Once that is set up I will email these two mailing lists with subscription information. My thoughts are to leverage the efforts of the GIN group and perhaps start with all agreements and SLA's being done by humans and verbal discussions - and see if we can actually use others cycles effectively using the existing technology before we leap into anything too complex.

hi Andrew, just a few comments. On Sa, 2010-10-30 at 18:01 +0200, Andrew Grimshaw wrote: [...]
Second, with respect to PGI
Application deployment/management. This came across as the number one requirement. This came as a surprise to me, though I agree it is an important next step to make execution services (OGSA-BES) more useful. There are at least two ways to handle this problem – one I’ll call manual installation, configuration, and reification (MICR), and the other I’ll call automatic deployment, configuration, and reification (ADCR).
MICR
This is definitely the easy case.
First, an application is deployed at/on a site/compute resource. Appropriate metadata is added to the resource metadata or an information service is updated with information on the local bindings for the executable, paths, libraries, etc.
Second, once a placement decision is made by a job manager (I’m using the OGSA-EMS term, here think meta-scheduler), the JSDL document that contains an abstract application name is reified (transformed) to have the site-appropriate values for the application.
Third, the JSDL is sent to the compute resource.
I believe this is basically how UNICORE 6 does this now.
in UNICORE the "reification" is done directly by the execution service, there is no need for a scheduler here.
ADCR
ADCR is similar to MICR except the deploying step of setting the application up and configuring the metadata is done automatically by a service. For simple applications – a single executable, or maybe a zip file that needs to be unzipped, this is easy. It can get complicated really fast. We at Genesis II defined a very simple deploying service combined with a reification service that worked for the easy case – executable and zip file. I can dig up the documentation on this if anyone is interested.
I believe that the general case for this is a tarpit for a standards body. I have a PhD student working on it J.
this use case is interesting, we've also thought about this. As you say, it gets complicated really fast :-) [...]
PGI profile [...]
My own interest is in extending the existing specs and working with GIN to drive spec modifications. If we go with extending the existing specs – BES in particular, should we modify BES just a little and leave the basic factory and management pattern the same? Or go with a cleaner model more consistent with what Bernd presented (return an activity handle <EPR?> and then manage the activity directly).
Here I have to make a correction. In fact the EMI people (specifically from ARC and gLite) do not (yet) seem to like the "manage activity directly" option, but want to manage all activities through a single manager. Specific activities are "addressed" by giving the activity ID as part of the messages. Sorry about this confusion. I'll raise this issue with the rest of the EMI folks when we discuss the concrete specification (wsdl/xsd). Personally I think EPRs, WS-Addressing and separate activities are the way to go. By the way, thanks again to you and the other PGI folks for the constructive feedback at OGF 30. Best regards, Bernd. -- Dr. Bernd Schuller Distributed Systems and Grid Computing Juelich Supercomputing Centre, http://www.fz-juelich.de/jsc Phone: +49 246161-8736 (fax -8556) Personal blog: www.jroller.com/page/gridhaus ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

hi Andrew, thanks for sharing your thoughts. i have just one comment this time :) On 2010-10-30 18:01, Andrew Grimshaw wrote:
PGI profile
This is the big question. We’ve gone through use cases, requirements, counting up the use cases that require each requirement, and two approaches to meeting those requirements (see slides).
How are we to proceed? Should we ask each proposer to provide more detail? Should we solicit more proposals? Should we work through both paths? Should we vote?
indeed, these are the difficult questions. these are the same questions the group faced before we went back to the use case/requirement collection/definition phase. whatever happens, the process should be transparent and sufficient time should be made available for group members to digest the developments and the process. cheers, Balazs -- Balázs Kónya Technical Director European Middleware Initiative www.eu-emi.eu NorduGrid Collaboration www.nordugrid.org Lund University balazs.konya@hep.lu.se Institute of Physics, EHEP phone: +46 46 222 8049 BOX 118, S - 221 00 LUND, Sweden fax: +46 46 222 4015

Balázs, and everyone else too, I agree that they are similar questions. I believe the situation is now different and improved. The difference is that when we started the use case and requirement definition phase we had proposals on how to proceed, e.g., the "Geneva Execution Service" and the "GES Realization via Existing Specifications" documents produced in the late Spring and Summer of 2009, but no solid requirements with which to compare the proposals. A regular refrain at that time was something along the line of "X does not meet our requirements", yet as a group we had no definition of the requirements. Trying to meet requirements in the absence of an agreement of exactly what the requirements are is a difficult challenge in any circumstance - even harder with such a diverse group. My recollection (perhaps flawed) is that is why we started the whole requirements process at the Munich meeting in March, so that we could make rational (or at least informed) decisions based on requirements going forward. Finally, I agree totally that the process must remain transparent and available to all members (or anyone who wants to join) to discuss. Andrew -----Original Message----- From: pgi-wg-bounces@ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf Of Balazs Konya Sent: Wednesday, November 03, 2010 5:37 AM To: pgi-wg@ogf.org Subject: Re: [Pgi-wg] Thoughts on the airplane home (GIN, PGI, Cycle sharing) hi Andrew, thanks for sharing your thoughts. i have just one comment this time :) On 2010-10-30 18:01, Andrew Grimshaw wrote:
PGI profile
This is the big question. Weve gone through use cases, requirements, counting up the use cases that require each requirement, and two approaches to meeting those requirements (see slides).
How are we to proceed? Should we ask each proposer to provide more detail? Should we solicit more proposals? Should we work through both paths? Should we vote?
indeed, these are the difficult questions. these are the same questions the group faced before we went back to the use case/requirement collection/definition phase. whatever happens, the process should be transparent and sufficient time should be made available for group members to digest the developments and the process. cheers, Balazs -- Balázs Kónya Technical Director European Middleware Initiative www.eu-emi.eu NorduGrid Collaboration www.nordugrid.org Lund University balazs.konya@hep.lu.se Institute of Physics, EHEP phone: +46 46 222 8049 BOX 118, S - 221 00 LUND, Sweden fax: +46 46 222 4015 _______________________________________________ Pgi-wg mailing list Pgi-wg@ogf.org http://www.ogf.org/mailman/listinfo/pgi-wg
participants (3)
-
Andrew Grimshaw
-
Balazs Konya
-
Bernd Schuller