April 2009 - pgi-wg - lists.ogf.org

Re: [Pgi-wg] EPR issue and approach using WS-Enumeration
by Andrew Grimshaw 08 Apr '09

08 Apr '09

This problem turned up early in RNS. We looked into WS-Enum and made a different choice (along with RNS). I am attaching a document that describes why we chose to go with a statefull iterator rather than enumeration,as well as all of the related documents. A > -----Original Message----- > From: pgi-wg-bounces(a)ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf Of > Morris Riedel > Sent: Wednesday, April 08, 2009 11:32 AM > To: pgi-wg(a)ogf.org > Subject: [Pgi-wg] EPR issue and approach using WS-Enumeration > > Hi PGI team, > > is there any experience by the members of using WS-Enumeration for > tackling with the emerging 'EPR pile' in the endpoint? > (WS-Enumeration is like separate google hits pages) > > I took a similar approach with Usage Records in OGF RUS a while ago since > this problem is nearly the same as ours - having scalability problems. > > > I think this is a problem where each middleware has a limit currently, > looking especially on emerging many task computing jobs (running rather > short - but many (potentially different) of them). > > (1) > Submitting 100.000 jobs and two fail. Both have to have a JSDL for > resubmit > - but this means you have typically 100.000 instances/EPRs tracked in the > container... > > (2) > Submitting 100.000 jobs and two fail using bulk submission or something > like > that with only one JSDL - you have difficulties to get the job description > of these two. > > > Maybe there is already someone having experience with WS-Enumeration in > this > context here in the PGI space? > > Thanks, > Morris > > > > ------------------------------------------------------------ > Morris Riedel > SW - Engineer > Distributed Systems and Grid Computing Division > Jülich Supercomputing Centre (JSC) > Forschungszentrum Juelich > Wilhelm-Johnen-Str. 1 > D - 52425 Juelich > Germany > > Email: m.riedel(a)fz-juelich.de > Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel > Phone: +49 2461 61 - 3651 > Fax: +49 2461 61 - 6656 > > Skype: MorrisRiedel > > "We work to better ourselves, and the rest of humanity" > > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe > Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), > Dr. Ulrich Krafft (stellv. Vorsitzender) >

2 1

EPR issue and approach using WS-Enumeration
by Morris Riedel 08 Apr '09

08 Apr '09

Hi PGI team, is there any experience by the members of using WS-Enumeration for tackling with the emerging 'EPR pile' in the endpoint? (WS-Enumeration is like separate google hits pages) I took a similar approach with Usage Records in OGF RUS a while ago since this problem is nearly the same as ours - having scalability problems. I think this is a problem where each middleware has a limit currently, looking especially on emerging many task computing jobs (running rather short - but many (potentially different) of them). (1) Submitting 100.000 jobs and two fail. Both have to have a JSDL for resubmit - but this means you have typically 100.000 instances/EPRs tracked in the container... (2) Submitting 100.000 jobs and two fail using bulk submission or something like that with only one JSDL - you have difficulties to get the job description of these two. Maybe there is already someone having experience with WS-Enumeration in this context here in the PGI space? Thanks, Morris ------------------------------------------------------------ Morris Riedel SW - Engineer Distributed Systems and Grid Computing Division Jülich Supercomputing Centre (JSC) Forschungszentrum Juelich Wilhelm-Johnen-Str. 1 D - 52425 Juelich Germany Email: m.riedel(a)fz-juelich.de Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel Phone: +49 2461 61 - 3651 Fax: +49 2461 61 - 6656 Skype: MorrisRiedel "We work to better ourselves, and the rest of humanity" Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender)

1 0

Re: [Pgi-wg] TelCon Service down?
by Moreno Marzolla 08 Apr '09

08 Apr '09

Morris Riedel wrote: > Me and Moreno can't join again... I get "call refused" :-| Moreno. -- Moreno Marzolla INFN Sezione di Padova, via Marzolo 8, 35131 PADOVA, Italy EMail: moreno.marzolla(a)pd.infn.it Phone: +39 049 8277103 WWW : http://www.dsi.unive.it/~marzolla Fax : +39 049 8756233

2 1

TelCon Service down?
by Morris Riedel 08 Apr '09

08 Apr '09

Me and Moreno can't join again... ------------------------------------------------------------ Morris Riedel SW - Engineer Distributed Systems and Grid Computing Division Jülich Supercomputing Centre (JSC) Forschungszentrum Juelich Wilhelm-Johnen-Str. 1 D - 52425 Juelich Germany Email: m.riedel(a)fz-juelich.de Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel Phone: +49 2461 61 - 3651 Fax: +49 2461 61 - 6656 Skype: MorrisRiedel "We work to better ourselves, and the rest of humanity" Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender)

1 0

Comments from Marvin Theimer June 5, 2006 (should be in archieve)
by Andrew Grimshaw 08 Apr '09

08 Apr '09

have the following questions about BES and the various discussions that have recently occurred (including the ESI integration): * Extensibility: * Given that BES has bought into the notion of an extensible activity state diagram, it needs to also normatively define how clients can learn of the extensions that a given BES service supports. Is that something that will be added to the BES specification? Or will the specification point to some other place where notions of extensibility are defined more generically? (Personally, I'd vote for the former approach.) * Is the "base case" for BES now fig.2, which shows states of {new, pending, running, canceled, failed, finished}? * Previously included states, such as Execution-Pending, will presumably be defined in suitable extension profiles? * Assuming that data staging and suspension are now extensions to the base BES spec, will they be defined as such in an appendix of the spec, or as a separate extension profile? * The original BES spec describes a fairly sophisticated data staging design that supports parallelism. Is there any interest in defining a second, simpler data staging extension that avoids the complexity of the parallelism support? * Will the suspension extension be the simple one that is currently presented in sec. 4 as an example? Or do people feel that a more complicated version, such as the ESI one is necessary/important? Can/should we define both? * Given that suspension is no longer in the base design, presumably the createInSuspendedState parameter to CreateActivityFromJSDL should disappear? * RequestActivityStateChange: I believe this operation will pose challenges in an extensible design. The current design is imperative by nature: it specifies an explicit state to move an activity to. However, a client who does not know of all the extensions that a BES service implements may not know how to pick the appropriate state to transition to. It seems better to introduce a more declarative approach in which clients specify "actions" they wish to occur, such as 'CancelActivity'. This approach would allow the BES service to make the appropriate state transition in response to a desired action requested by a client. **** Comment by Andrew on April 8, 2009 In Genesis II we have implemented sub-states for staging (in, out) and suspend/resume. I'm thinking that we should profile the substates, and add change state. A

1 0

Re: [Pgi-wg] OGF PGI - Security Model - NEW versions of GSI acceptRFC-3820-compliant X509 proxies
by Vincenzo Ciaschini 08 Apr '09

08 Apr '09

Also note that a change in configuration, as described in my previous mail, also allows the current VOMS server to be contacted correctly and having a rfc proxy as the final result. Ciao, Vincenzo Morris Riedel wrote: > Hi Duane, > >> -Morris: What is the chance that this VOMS 2.0 get a huge deployment in > EGEE then?! > > It doesn't need a huge deployment. Even a single deployment at the grid > boundary will work to our ends. > > I don’t think so for production interop – maybe for typical OGF interop > fests. But not for production applications, which are planned between DEISA > and EGEE for a few use cases… > > I guess one VOMS is deployed for each VO or has this changed? > > So it depends with the VO you are working with, e.g. FUSION interop > application and their deployed VOMS service in EGEE for instance will matter > for DEISA… (EUFORIA work I’m involved) > > The difference to OGF interop tests is that we would like to use the interop > in production – which is different than deploying somewhere a VOMS for > special needs. However, it might be the only solution if PGI is not coming > up with agreements… :-) > > Take care, > Morris

1 0

Re: [Pgi-wg] OGF PGI - GES telephone conference today - Parameter Sweep Extension
by Etienne URBAH 08 Apr '09

08 Apr '09

Morris, Concerning the OGF PGI - GES telephone conference today : Desktop Grids, which provide large numbers of small computers, are very interested by 'Parameter Sweep Extension', which should permit to send in only 1 step a large number of jobs with same executable, but different parameters. Is it in the scope of today's discussion ? If yes, thank you for taking it into account. Best regards. ---------------------------------- Etienne URBAH IN2P3 - LAL Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Mob: +33 6 22 30 53 27 Skype: etienne.urbah mailto:urbah@lal.in2p3.fr ---------------------------------- On Wed, 8 Apr 2009, Morris Riedel wrote: > Dear PGI team, > > > Moreno will chair a telcon about the PGI GES Strawman document today! > > > 2009-04-08 - 4 pm CET > > Call-in details as follow: > via Skype call +9900827049931906 (free of charge) ordinary phone numbers > (local rates) with the 9931906 conference number: > > Austria 0820 401 15470 > Belgium 0703 57 134 > France 0826 109 071 > Germany +49 (0) 180 500 9527 > Switzerland 0848 560 397 > > > Proposed agenda: > > > (1) > GES Strawman document > > > (2) > AOB > > > > Material can be found here: > > http://forge.gridforum.org/sf/docman/do/listDocuments/projects.pgi-wg/docman > .root.input_documents.execution_service > > # Strawman Document > # Initial supportive slides about some strawman content > > > > > Your co-chair, > Morris > > > ------------------------------------------------------------ > Morris Riedel > SW - Engineer > Distributed Systems and Grid Computing Division > Jülich Supercomputing Centre (JSC) > Forschungszentrum Juelich > Wilhelm-Johnen-Str. 1 > D - 52425 Juelich > Germany > > Email: m.riedel(a)fz-juelich.de > Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel > Phone: +49 2461 61 - 3651 > Fax: +49 2461 61 - 6656 > > Skype: MorrisRiedel

3 2

Re: [Pgi-wg] OGF PGI - Security Model - NEW versions of GSI acceptRFC-3820-compliant X509 proxies
by Duane Merrill 08 Apr '09

08 Apr '09

*Steven*: Surely its better to focus our energies on defining a profile around the new style proxies that groups intend to support going forward? This seems most prudent. We don't need to bend the PGI profile(s) such that every existing endpoint can be labeled "compliant". Compliant service endpoints can be rolled out incrementally as subject to implementation/budget/etc. constraints. *Morris: *What is the chance that this VOMS 2.0 get a huge deployment in EGEE then?! It doesn't need a huge deployment. Even a single deployment at the grid boundary will work to our ends. -Duane 2009/4/8 Morris Riedel <m.riedel(a)fz-juelich.de> > Hi, > > very valuable information - probably another reason for sticking to GSI > unfortunately in the production space... > > >- VOMS 2.0 is due to be out during autumn this year. > > What is the chance that this VOMS 2.0 get a huge deployment in EGEE then?! > > Thanks, > Morris > > ------------------------------------------------------------ > Morris Riedel > SW - Engineer > Distributed Systems and Grid Computing Division > Jülich Supercomputing Centre (JSC) > Forschungszentrum Juelich > Wilhelm-Johnen-Str. 1 > D - 52425 Juelich > Germany > > Email: m.riedel(a)fz-juelich.de > Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel > Phone: +49 2461 61 - 3651 > Fax: +49 2461 61 - 6656 > > Skype: MorrisRiedel > > "We work to better ourselves, and the rest of humanity" > > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe > Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), > Dr. Ulrich Krafft (stellv. Vorsitzender) > > > >------Original Message----- > >-From: pgi-wg-bounces(a)ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf > Of > >-Vincenzo Ciaschini > >-Sent: Wednesday, April 08, 2009 12:07 PM > >-To: Etienne URBAH > >-Cc: aleksandr.konstantinov(a)fys.uio.no; edges-na3(a)mail.edges-grid.eu; > >-lodygens(a)lal.in2p3.fr; pgi-wg(a)ogf.org > >-Subject: Re: [Pgi-wg] OGF PGI - Security Model - NEW versions of GSI > acceptRFC- > >-3820-compliant X509 proxies > >- > >-Hi Etienne, > >-Etienne URBAH wrote: > >-> Still to be verified is that VOMS servers only accept GSI-style X509 > >-> proxies http://forge.gridforum.org/sf/go/doc15591?nav=1 > >-VOMS accepts and generates both type of proxies. However, there is a > >-caveat, which explains the failures you get: > >- > >-Pre VOMS 2.0: > >-Server-side, VOMS uses GSI for validation. This means that if you run > >-voms against gt2, contacting it with a gt4 proxy will fail. > >- > >-There is a final argument in the vomses file which specifies which > >-version of GT the service uses, and adapts the proxies used to contact > >-it accordingly. Many VOs distribute an incorrect vomses file. > >- > >-The final proxy obtained as output by voms-proxy-init will always be > >-what you requested, in this case a rfc proxy. > >- > >-VOMS 2.0 onwards: > >-Globus dependencies on the server will be dropped too (They are > >-corrently removed from both the clients and the APIs). This will mean > >-that any kind of proxy, or even a bare certificate, will become > >-acceptable for contacting the service. The whole vomses config business > >-above will no longer be relevant. > >- > >-VOMS 2.0 is due to be out during autumn this year. > >- > >-Ciao, > >- Vincenzo > >-_______________________________________________ > >-Pgi-wg mailing list > >-Pgi-wg(a)ogf.org > >-http://www.ogf.org/mailman/listinfo/pgi-wg > > _______________________________________________ > Pgi-wg mailing list > Pgi-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/pgi-wg > >

2 1

Re: [Pgi-wg] TELCON Numbers and Reminder: 4 pm CET TODAY: GESDiscussions
by Andrew Grimshaw 08 Apr '09

08 Apr '09

All, I have read the document Moreno provided very carefully and have a large number of comments that I will share when we discuss the document morning/afternoon. First, thank you Moreno for putting "on paper". My high level comments are: 1) I that I feel it is important to clearly separate requirements from mechanism in the first discussion. There many ways to solve a problem. 2) That minimalism in composable porttypes leads to cleaner design. In particular, it might make sense to define a simple ActivityEndpoint porttype that provides information for example on uri's to access the execution directory for staging, etc. To quote another European "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." Antoine de Saint-Exupery 3) The differences from the current BES are pretty minor, e.g., vectorization of create and change state. As an aside, our experience with the current BES specification is that passing back the list of activities "contained" by a BES resource is awkward at best, and a performance inhibitor at worst. A direct call to get the instances makes much more sense. They way we have done this -- that does not require any new specification is via the use of the RNS porttype to "list" activities. Andrew Ps. Moreno, it might be useful if you used a screen sharing software such as glance while going through the document. > -----Original Message----- > From: pgi-wg-bounces(a)ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf Of > Morris Riedel > Sent: Wednesday, April 08, 2009 7:40 AM > To: pgi-wg(a)ogf.org > Subject: [Pgi-wg] TELCON Numbers and Reminder: 4 pm CET TODAY: > GESDiscussions > > Dear PGI team, > > > Moreno will chair a telcon about the PGI GES Strawman document today! > > > 2009-04-08 - 4 pm CET > > Call-in details as follow: > via Skype call +9900827049931906 (free of charge) ordinary phone numbers > (local rates) with the 9931906 conference number: > > Austria 0820 401 15470 > Belgium 0703 57 134 > France 0826 109 071 > Germany +49 (0) 180 500 9527 > Switzerland 0848 560 397 > > > Proposed agenda: > > > (1) > GES Strawman document > > > (2) > AOB > > > > Material can be found here: > > http://forge.gridforum.org/sf/docman/do/listDocuments/projects.pgi- > wg/docman > .root.input_documents.execution_service > > # Strawman Document > # Initial supportive slides about some strawman content > > > > > Your co-chair, > Morris > > > ------------------------------------------------------------ > Morris Riedel > SW - Engineer > Distributed Systems and Grid Computing Division > Jülich Supercomputing Centre (JSC) > Forschungszentrum Juelich > Wilhelm-Johnen-Str. 1 > D - 52425 Juelich > Germany > > Email: m.riedel(a)fz-juelich.de > Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel > Phone: +49 2461 61 - 3651 > Fax: +49 2461 61 - 6656 > > Skype: MorrisRiedel > > "We work to better ourselves, and the rest of humanity" > > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe > Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), > Dr. Ulrich Krafft (stellv. Vorsitzender) >

1 1

Re: [Pgi-wg] Supportive slides: Request routing idea addressed
by Steven Newhouse 08 Apr '09

08 Apr '09

> It provides more information about the 'Request routing' idea addressed > by UNICORE/ARC/gLite... Since when is this a core requirement for dealing with batch submission to a site? Is this not out of scope? You are opening up a can of worms regarding complex workflows etc. Steven

2 1