October 2010 - pgi-wg - lists.ogf.org

Thoughts on the airplane home (GIN, PGI, Cycle sharing)
by Andrew Grimshaw 07 Nov '10

07 Nov '10

All, I thought we had a very productive OGF. A number of interesting and exciting results in GIN, and solid progress on PGI requirements and a discussion of how to meet those requirements going forward. A couple of things stuck in my mind - so I thought I'd get them out there and get a response. First, with respect to GIN. I thought the demo's, particularly the use of the BES endpoints in SAGA, were very informative. Particularly when one keeps in mind that at least the Unicore/futuregrid resources were not even available until Friday afternoon (eastern time!). I agree with the sentiment that we should build on this very successful demo. In particular most groups have either done simple extensions to JSDL, or used existing mechanism in a particular way. Ditto with BES factory attributes. Several ideas came up (Morris, chime in I think you were keeping notes) Profiling we can do fairly easily: File system JSDL specifies how file systems can be defined - Mark correct me on this. We use one we call SCRATCH as a place to cache files that are large and constant, including often zip files that contain applications. We tell JSDL to mount it, we then use the "don't overwrite on stage-in" JSDL flag, and the "don't delete when you're done" flag. This is a huge help. Extra attributes We have found a need for several extra BES factory attributes and the use of those in JSDL matching. I'm guessing many of you have done this as well. These include whether an application is installed, whether static linking is required, I also think that while we are waiting for PGI to figure out how to deploy applications that we have a simple convention for applications, something like HAS_APPX_INSTALLED And then if the BES factory has that, then by convention that APPX_PATH and APPX_LIB_PATH are defined and can be used in some sort of prefix. File staging - demo of sites that support FSE at SC. We discussed that several implementations support FSE. Let's test that interop. It really changes the set of applications you can run if you can copy them onto the machine. If we can get a set of endpoints that support FSE we'll (GenesisII/XCG) set up a meta-scheduler that takes JSDL and manage them through the life cycle on the endpoints of the different implementations .This might also form the basis for some cycle trading ... Re: FutureGrid. I also want to repeat that if groups want to put up standards based implementations of services on FutureGrid (OGSA-BES, RNS, WS-Trust, ByteIO, gridFTP, SRM) we will work with them to make it so. One of the goals of FutureGrid is to provide PERSISTENT standards based endpoints with which to test. Second, with respect to PGI Application deployment/management. This came across as the number one requirement. This came as a surprise to me, though I agree it is an important next step to make execution services (OGSA-BES) more useful. There are at least two ways to handle this problem - one I'll call manual installation, configuration, and reification (MICR), and the other I'll call automatic deployment, configuration, and reification (ADCR). MICR This is definitely the easy case. First, an application is deployed at/on a site/compute resource. Appropriate metadata is added to the resource metadata or an information service is updated with information on the local bindings for the executable, paths, libraries, etc. Second, once a placement decision is made by a job manager (I'm using the OGSA-EMS term, here think meta-scheduler), the JSDL document that contains an abstract application name is reified (transformed) to have the site-appropriate values for the application. Third, the JSDL is sent to the compute resource. I believe this is basically how UNICORE 6 does this now. ADCR ADCR is similar to MICR except the deploying step of setting the application up and configuring the metadata is done automatically by a service. For simple applications - a single executable, or maybe a zip file that needs to be unzipped, this is easy. It can get complicated really fast. We at Genesis II defined a very simple deploying service combined with a reification service that worked for the easy case - executable and zip file. I can dig up the documentation on this if anyone is interested. I believe that the general case for this is a tarpit for a standards body. I have a PhD student working on it J. **** A note on reification >From Wikipedia Reification generally refers to bringing into being or turning concrete. Specifically, reification may refer to: * Reification (computer science) <http://en.wikipedia.org/wiki/Reification_%28computer_science%29> , making a data model for a previously abstract concept In this context I refer to taking an abstract JSDL that may give an abstract name of an application, e.g., <JSDL-abstract-application-name>/bin/biology/blastp-v5 </ JSDL-abstract-application-name> THIS IS MADE UP SYNTAX That might occur in a JSDL file, and then making it concrete for a particular endpoint by transforming the JSDL to have the actual local path name and defining any environment variables needed. We discussed this problem extensively in the OGSA, OGSA-EMS, and OGSA-BES working groups. A strawman interface was even defined. MPI Once again the importance of this came as a somewhat welcome surprise. I believe this is of considerable interest to UNICORE. Perhaps they could take the lead on this one. Is it simply a problem of the JSDL not being expressive enough to both select appropriate resources and pass topology information down to MPI? In other words, is this an easy problem? Or is there something else I am not seeing? Security The single greatest impediment to the GIN demonstrations was working out the security stuff. Andre was pretty clear about this. In other words, the functional aspects of the standards were not the problem - the certificates and other stuff were. It has been suggested that it may be time to dive back into this. We spent quite a bit of time on it 18 months ago. The result was a DRAFT profile document by Duane Merrill that bifurcated the profile into a transport level security profile based on delegated X.509 certificates and a message level security profile - primarily aimed at SAML (http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi-wg/do cman.root.input_documents.security_material.comm_profile/doc15575) . I know that there was a lively debate about how to proceed. PGI profile This is the big question. We've gone through use cases, requirements, counting up the use cases that require each requirement, and two approaches to meeting those requirements (see slides). How are we to proceed? Should we ask each proposer to provide more detail? Should we solicit more proposals? Should we work through both paths? Should we vote? My own interest is in extending the existing specs and working with GIN to drive spec modifications. If we go with extending the existing specs - BES in particular, should we modify BES just a little and leave the basic factory and management pattern the same? Or go with a cleaner model more consistent with what Bernd presented (return an activity handle <EPR?> and then manage the activity directly). Either way we need make some decisions and move forward. Third, with respect to Cycle sharing At first I thought that the cycle sharing BOF did not lead to any interested in trying this out. As the week wore on my initial impression proved mistaken. Johannes followed up with several other groups and we may indeed have a sufficient critical mass to proceed. I will ask Joel to set up a source forge site and email list. Once that is set up I will email these two mailing lists with subscription information. My thoughts are to leverage the efforts of the GIN group and perhaps start with all agreements and SLA's being done by humans and verbal discussions - and see if we can actually use others cycles effectively using the existing technology before we leap into anything too complex.

3 3

PGI requirements list
by Johannes Watzl 28 Oct '10

28 Oct '10

Dear all, we are just finished with the PGI sessions here at OGF30. The requirements list including a prioritization based on the use cases can be found on GridForge: http://forge.gridforum.org/sf/go/doc16080?nav=1 If you have any objections, please send a message to the list before the next PGI call on Thursday, 4 November 2010. Best, Johannes -- _ _ _ _ _ _ Johannes Watzl |\/| |\ | |\/| Institut für Informatik / Dept. of CS | | | \| | | Ludwig-Maximilians-Universität München ======= TEAM ======= Oettingenstr. 67, 80538 Munich, Germany Room E 005, Phone +49-89-2180-9162 Munich Network Management Team Email: watzl(a)nm.ifi.lmu.de Münchner Netz-Management Team http://www.nm.ifi.lmu.de/~watzl

4 3

about expressing requirements on resources in JSDL using XQuery
by Sergio Andreozzi 26 Oct '10

26 Oct '10

Hi Morris, this document contains examples for the idea of using XQuery to express requirement on an XML-based resource description (in this case this is GLUE 2.0) http://ogf.org/documents/GFD.137.pdf The advantage of this approach is that JSDL is not bound to a set of terms and does not need to be changed according to the evolution of the information model. Cheers, Sergio -- Sergio Andreozzi - Policy Development Manager EGI.eu - Amsterdam - The Netherlands - http://egi.eu

1 0

Update TelCon Minutes for Discussed Requirements
by Morris Riedel 22 Oct '10

22 Oct '10

All, I have updated all the telcon minutes as promised: http://forge.gridforum.org/sf/go/projects.pgi-wg/discussion.meetings The already discussed and tuned descriptions of requirements should be taken into account when presenting at OGF next week! CU in Brussels. Take care, Morris ------------------------------------------------------------ Morris Riedel Deputy Division Leader Distributed Systems and Grid Computing Division Jülich Supercomputing Centre (JSC) Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel

1 0

Reminder & Agenda: PGI Telcon 2010-10-21; 3:30 pm CET
by Morris Riedel 21 Oct '10

21 Oct '10

Dear PGI team, just a gentle reminder about our usual telcon. (1) Telcon-System and Screen-Sharing http://webconf.vc.dfn.de/ogf-pgi-wg/ No login is required. Just enter as 'guest' with your real world name. (2) Proposed agenda: (a) Debrief notes from the last meeting Reminder for use case/requirements presenters at OGF30! (b) Discussions on job descriptions/management based on use cases Preparation and prioritization for specification & OGF30 presentations. (c) AOB Please feel free to add any other topic on the agenda that I may have missed. Your co-chair, Morris ------------------------------------------------------------ Morris Riedel Deputy Division Leader Distributed Systems and Grid Computing Division Jülich Supercomputing Centre (JSC) Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel

1 0

OGF PGI - Requirements for Job Description
by Etienne URBAH 19 Oct '10

19 Oct '10

Morris, Johannes and all, Concerning the OGF PGI requirements for Job Description : From the EDGI use case 'Marshal Activities from Service Grid to Desktop Grid', the most important requirements, sorted by decreasing priority, are : JD3 (100) The Job Description document MUST reference all grid entities in conformance to the GLUE specification JD11 (107) Job Description should be only used for Job Description not for any kind of feedback. Job status information should not be communicated with a 'changed JSDL' JD4 (101) The Job Description document specification MUST permit the Client to request automatic data stage-in and stage-out JD22 (116) Clearly separate file and directory in the data-stagings. e.g. within the job description the element should be clearly labeled as file or directory, e.g. different aspect of handling directory or a file The other requirements labeled 'yes' are welcome, EXCEPT the 2 following ones, which generate too much COMPLEXITY in the Execution Service : JD6 (103) The Job Description document specification MUST permit the Client to request that at specified Activity states, the Execution service sets the Activity on 'Hold' [NO] JD8 (105) Job Description supports, 1) service-directed data-staging (i.e. specify data-stagings),2) client-initiated data-staging (i.e. specifiy data-staging but service will do nothing only control whether files existing), 3) manual data-staging (i.e. no specification of data-staging elements) [NO] Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------

1 3

Requirements mapping: compute area
by Oxana Smirnova 14 Oct '10

14 Oct '10

Hi all, I'm catching up on requirements mapping; so here is the list of compute area requirements that map (IMO) to the ARC, EGI and GROMACS use cases: ARC ----- 34, 35, 43, 44, 45, 46, 56, 57, 63, 64, 66, 68, 76, 78, 80, 81, 82, 85, 86, 90, 94, 100, 101, 104, 105, 108, 112, 114, 116, 119, 126, 133, 141, 143, 144 EGI ----- 34, 35, 37, 43, 44, 45, 46, 56, 57, 61, 62, 63, 64, 65, 66, 68, 72, 76, 78, 80, 81, 82, 84, 86, 90, 94, 100, 101, 104, 105, 108, 112, 114, 119, 126, 133, 141, 143, 144 GROMACS -------------- 34, 35, 43, 44, 45, 56, 57, 61, 63, 34, 66, 68, 76, 78, 81, 82, 86, 90, 101, 104, 105, 108, 112, 116, 119, 143, 144 Cheers, Oxana

3 2

Reminder & Agenda: PGI Telcon 2010-10-14; 3:30 pm CET
by Morris Riedel 14 Oct '10

14 Oct '10

Dear PGI team, just a gentle reminder about our usual telcon. (1) Telcon-System and Screen-Sharing http://webconf.vc.dfn.de/ogf-pgi-wg/ No login is required. Just enter as 'guest' with your real world name. (2) Proposed agenda: (a) Debrief notes from the last meeting Reminder for use case/requirements presenters at OGF30! (b) Discussions on job descriptions based on use cases Preparation and prioritization for specification & OGF30 presentations. (c) AOB Please feel free to add any other topic on the agenda that I may have missed. Your co-chair, Morris ------------------------------------------------------------ Morris Riedel Deputy Division Leader Distributed Systems and Grid Computing Division Jülich Supercomputing Centre (JSC) Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel

1 0

Requirements mapping: Continued
by Oxana Smirnova 14 Oct '10

14 Oct '10

Hi again, here are the rest of requirements mapped to ARC, EGI and GROMACS use cases: ARC ----- Info: 1, 4, 6 Security: 9, 28, 29, 30, 32, 11 NF: 145, 146, 148, 149, 150, 156, 160 EGI ---- Accounting: 36 Info: 1, 4, 6 Security: 9, 17, 20, 21, 22, 23, 28, 29, 30, 32, 11 NF: 145, 146, 147, 148, 149, 150, 156, 160 GROMACS ------------- Info:1, 4, 6 SecurityL 28, 29, 11 NF: 148, 149, 156 I have to note that I did not prioritize requirements, because they can not be compared: there is no way to argue that e.g. a requirement of traceability is more or less important than that of using SOAP. I also should say that I noticed no duplicates, at least no obvious ones. And, the level of requirements varies really a lot, from extremely generic to extremely specific, which again prevents comparison. Cheers, Oxana

1 0

PGI Use Cases: OGF Editor Submission
by Morris Riedel 07 Oct '10

07 Oct '10

Dear PGI team, we just submitted the PGI use cases 1.0 document to the OGF editor (Greg Newby). The version submitted can be found here: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi-wg/do… So it will be in public comment soon too. Your co-chairs, Morris & Johannes ------------------------------------------------------------ Morris Riedel Deputy Division Leader Distributed Systems and Grid Computing Division Jülich Supercomputing Centre (JSC) Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel

1 0