Update on PGI Action (Move spec to drafts)
Hi team, here another update on one action item:
- A - action (Morris): copy strawman rendering doc to drafts
Done. The actual draft specification (version 0.37) is now accessible in GridForge via: Doucments - Root Folder - Drafts - Specification Direct link: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi-wg/doc... So far, I kept a copy within the Input Document Section since I propose a renaming of the document within drafts to: "Draft PGI Specification" It is neither a strawman document nor a concrete rendering - it's just a draft specification from PGI. However, of course, I seek to discuss this tomorrow in the telcon with all of you. Your co-chair, Morris ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
Hi team, note that somehow Grid Forge provides a "zip" file instead of a "odt" via my provided URL - but if you rename it to ".odt" again it's fine and it can be opened with OpenOffice. I discuss a better solution with some folks that work typically with open office... Just 4 Info, Morris ________________________________________ Von: pgi-wg-bounces@ogf.org [pgi-wg-bounces@ogf.org] im Auftrag von Riedel, Morris [m.riedel@fz-juelich.de] Gesendet: Donnerstag, 17. Dezember 2009 10:37 An: pgi-wg@ogf.org Betreff: [Pgi-wg] Update on PGI Action (Move spec to drafts) Hi team, here another update on one action item:
- A - action (Morris): copy strawman rendering doc to drafts
Done. The actual draft specification (version 0.37) is now accessible in GridForge via: Doucments - Root Folder - Drafts - Specification Direct link: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi-wg/doc... So far, I kept a copy within the Input Document Section since I propose a renaming of the document within drafts to: "Draft PGI Specification" It is neither a strawman document nor a concrete rendering - it's just a draft specification from PGI. However, of course, I seek to discuss this tomorrow in the telcon with all of you. Your co-chair, Morris ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ _______________________________________________ Pgi-wg mailing list Pgi-wg@ogf.org http://www.ogf.org/mailman/listinfo/pgi-wg
Morris and all, I have updated the 'Draft PGI Specification' with version 0.38 at http://forge.gridforum.org/sf/go/doc15839?nav=1 with following improvements : - Exclude internal states. - Second level Job states are also mandatory. - Inside the 'Submitted' state : - The Execution Service may already allocate resources, - Rename 'Waiting' substate as 'Validated', - Add 'Hold' substate. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Thu, 17 Dec 2009, Riedel, Morris wrote:
Hi team,
here another update on one action item:
- A - action (Morris): copy strawman rendering doc to drafts
Done.
The actual draft specification (version 0.37) is now accessible in GridForge via: Doucments - Root Folder - Drafts - Specification Direct link: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi-wg/doc...
So far, I kept a copy within the Input Document Section since I propose a renaming of the document within drafts to:
"Draft PGI Specification"
It is neither a strawman document nor a concrete rendering - it's just a draft specification from PGI.
However, of course, I seek to discuss this tomorrow in the telcon with all of you.
Your co-chair, Morris
------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
All, Some good progress seems to be being made on the specification and the draft is beginning to become coherent. A few thoughts emerged following a quick read through. * ChangeActivityState This reminded me of discussions we had in the late stages of BES. A similar operation was moved in & out of the draft before finally being removed. There was discomfort as to the amount of influence an end-user agent could have on the underlying state machine. A user could not force a job to execute if it was queued if there were not processes available. With the expanded state model that is now proposed there is more scope for user control - fine. You have the notion of legal and illegal state change requests but I could not see these documented on the state diagram. I would certainly propose changing this to 'RequestActivityStateChange' to make it clear that this is an initiation and that it make take some time for the underlying state engine to change. You effectively acknowledge this in your estimated time to change return value. I can see it being useful to have a 'this has already been done' option in the return body. I'm not sure how useful any value other than zero effectively is here. To me you are effectively saying this has either been done or you need to set up a notification or a poll to check back to see when it has completed. The estimate itself does not seem to be that valuable... * Partitioning The three port types all seem sensibly scoped. I have great concerns about the AGU-JSDL section. I'm not sure if this is just a place holder at the moment. I would be VERY concerned if this AGU-JSDL forks away from the JSDL activity. JSDL has gained considerable traction and is being further developed in the JSDL 2.0 discussions. I would have thought PGI would be better placed supporting a MINIMAL extension to JSDL to support its immediate needs, and feeding the other improvements it could see being needed into the broader JSDL discussions. * Delegation & Information PortTypes This must have broader use than just beyond PGI. If there are opportunities to do this why not put these into separate specifications even if they are still done through the PGI-WG. Cheers, Steven Dr Steven Newhouse EGEE Technical Director Interim EGI.eu Director Contact Details: http://cern.ch/Steven.Newhouse
-----Original Message----- From: pgi-wg-bounces@ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf Of Etienne URBAH Sent: 17 December 2009 23:33 To: Morris Riedel; pgi-wg@ogf.org Cc: edges-na3@mail.edges-grid.eu; lodygens@lal.in2p3.fr Subject: Re: [Pgi-wg] OGF PGI : Draft PGI Specification
Morris and all,
I have updated the 'Draft PGI Specification' with version 0.38 at http://forge.gridforum.org/sf/go/doc15839?nav=1 with following improvements :
- Exclude internal states.
- Second level Job states are also mandatory.
- Inside the 'Submitted' state : - The Execution Service may already allocate resources, - Rename 'Waiting' substate as 'Validated', - Add 'Hold' substate.
Best regards.
----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------
On Thu, 17 Dec 2009, Riedel, Morris wrote:
Hi team,
here another update on one action item:
- A - action (Morris): copy strawman rendering doc to drafts
Done.
The actual draft specification (version 0.37) is now accessible in GridForge via: Doucments - Root Folder - Drafts - Specification Direct link:
wg/docman.root.drafts.specification/doc15839
So far, I kept a copy within the Input Document Section since I
http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi- propose a renaming of the document within drafts to:
"Draft PGI Specification"
It is neither a strawman document nor a concrete rendering - it's
just a draft specification from PGI.
However, of course, I seek to discuss this tomorrow in the telcon
with all of you.
Your co-chair, Morris
---------------------------------------------------------------------
-------------------------- ---------------------------------------------------------------------
-------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ---------------------------------------------------------------------
-------------------------- ---------------------------------------------------------------------
- - - -
--------------------------
Hi Steven, thanks for your valuable comments - I put their discussion on the agenda for today. Take care, Morris ________________________________________ Von: pgi-wg-bounces@ogf.org [pgi-wg-bounces@ogf.org] im Auftrag von Steven Newhouse [Steven.Newhouse@cern.ch] Gesendet: Freitag, 18. Dezember 2009 09:44 An: pgi-wg@ogf.org Betreff: Re: [Pgi-wg] OGF PGI : Draft PGI Specification All, Some good progress seems to be being made on the specification and the draft is beginning to become coherent. A few thoughts emerged following a quick read through. * ChangeActivityState This reminded me of discussions we had in the late stages of BES. A similar operation was moved in & out of the draft before finally being removed. There was discomfort as to the amount of influence an end-user agent could have on the underlying state machine. A user could not force a job to execute if it was queued if there were not processes available. With the expanded state model that is now proposed there is more scope for user control - fine. You have the notion of legal and illegal state change requests but I could not see these documented on the state diagram. I would certainly propose changing this to 'RequestActivityStateChange' to make it clear that this is an initiation and that it make take some time for the underlying state engine to change. You effectively acknowledge this in your estimated time to change return value. I can see it being useful to have a 'this has already been done' option in the return body. I'm not sure how useful any value other than zero effectively is here. To me you are effectively saying this has either been done or you need to set up a notification or a poll to check back to see when it has completed. The estimate itself does not seem to be that valuable... * Partitioning The three port types all seem sensibly scoped. I have great concerns about the AGU-JSDL section. I'm not sure if this is just a place holder at the moment. I would be VERY concerned if this AGU-JSDL forks away from the JSDL activity. JSDL has gained considerable traction and is being further developed in the JSDL 2.0 discussions. I would have thought PGI would be better placed supporting a MINIMAL extension to JSDL to support its immediate needs, and feeding the other improvements it could see being needed into the broader JSDL discussions. * Delegation & Information PortTypes This must have broader use than just beyond PGI. If there are opportunities to do this why not put these into separate specifications even if they are still done through the PGI-WG. Cheers, Steven Dr Steven Newhouse EGEE Technical Director Interim EGI.eu Director Contact Details: http://cern.ch/Steven.Newhouse
-----Original Message----- From: pgi-wg-bounces@ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf Of Etienne URBAH Sent: 17 December 2009 23:33 To: Morris Riedel; pgi-wg@ogf.org Cc: edges-na3@mail.edges-grid.eu; lodygens@lal.in2p3.fr Subject: Re: [Pgi-wg] OGF PGI : Draft PGI Specification
Morris and all,
I have updated the 'Draft PGI Specification' with version 0.38 at http://forge.gridforum.org/sf/go/doc15839?nav=1 with following improvements :
- Exclude internal states.
- Second level Job states are also mandatory.
- Inside the 'Submitted' state : - The Execution Service may already allocate resources, - Rename 'Waiting' substate as 'Validated', - Add 'Hold' substate.
Best regards.
----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------
On Thu, 17 Dec 2009, Riedel, Morris wrote:
Hi team,
here another update on one action item:
- A - action (Morris): copy strawman rendering doc to drafts
Done.
The actual draft specification (version 0.37) is now accessible in GridForge via: Doucments - Root Folder - Drafts - Specification Direct link:
wg/docman.root.drafts.specification/doc15839
So far, I kept a copy within the Input Document Section since I
http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi- propose a renaming of the document within drafts to:
"Draft PGI Specification"
It is neither a strawman document nor a concrete rendering - it's
just a draft specification from PGI.
However, of course, I seek to discuss this tomorrow in the telcon
with all of you.
Your co-chair, Morris
---------------------------------------------------------------------
-------------------------- ---------------------------------------------------------------------
-------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ---------------------------------------------------------------------
-------------------------- ---------------------------------------------------------------------
- - - -
--------------------------
Pgi-wg mailing list Pgi-wg@ogf.org http://www.ogf.org/mailman/listinfo/pgi-wg
------Original Message----- -From: pgi-wg-bounces@ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf Of Steven Newhouse -Sent: Friday, December 18, 2009 12:44 PM -To: pgi-wg@ogf.org -Subject: Re: [Pgi-wg] OGF PGI : Draft PGI Specification - -All, - -Some good progress seems to be being made on the specification and the draft is beginning to become coherent. A few thoughts emerged -following a quick read through. - -* ChangeActivityState -This reminded me of discussions we had in the late stages of BES. A similar operation was moved in & out of the draft before finally being -removed. There was discomfort as to the amount of influence an end-user agent could have on the underlying state machine. A user could not -force a job to execute if it was queued if there were not processes available. With the expanded state model that is now proposed
-scope for user control - fine. You have the notion of legal and illegal state change requests but I could not see these documented on the state -diagram. I would certainly propose changing this to 'RequestActivityStateChange' to make it clear that this is an initiation and
-some time for the underlying state engine to change. You effectively acknowledge this in your estimated time to change return value. - -I can see it being useful to have a 'this has already been done' option in the return body. I'm not sure how useful any value other than zero -effectively is here. To me you are effectively saying this has either been done or you need to set up a notification or a poll to check back to see -when it has completed. The estimate itself does not seem to be that valuable... - -* Partitioning -The three port types all seem sensibly scoped. I have great concerns about the AGU-JSDL section. I'm not sure if this is just a
Steven, all, first and foremost thank you for your valuable comments Steven. I put your comments in the working slides to be more effectively discussed and tracked. The discussion of this e-mail is on the agenda today. Take care, Morris ------------------------------------------------------------ Morris Riedel Jülich Supercomputing Centre (JSC) Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel "We work to better ourselves, and the rest of humanity" Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender) there is more that it make take place holder at
-the moment. I would be VERY concerned if this AGU-JSDL forks away from the JSDL activity. JSDL has gained considerable traction and is -being further developed in the JSDL 2.0 discussions. I would have thought PGI would be better placed supporting a MINIMAL extension to -JSDL to support its immediate needs, and feeding the other improvements it could see being needed into the broader JSDL discussions. - -* Delegation & Information PortTypes -This must have broader use than just beyond PGI. If there are opportunities to do this why not put these into separate specifications even if -they are still done through the PGI-WG. - -Cheers, - -Steven - -Dr Steven Newhouse -EGEE Technical Director -Interim EGI.eu Director -Contact Details: http://cern.ch/Steven.Newhouse - - -> -----Original Message----- -> From: pgi-wg-bounces@ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf -> Of Etienne URBAH -> Sent: 17 December 2009 23:33 -> To: Morris Riedel; pgi-wg@ogf.org -> Cc: edges-na3@mail.edges-grid.eu; lodygens@lal.in2p3.fr -> Subject: Re: [Pgi-wg] OGF PGI : Draft PGI Specification -> -> Morris and all, -> -> I have updated the 'Draft PGI Specification' with version 0.38 at -> http://forge.gridforum.org/sf/go/doc15839?nav=1 with following -> improvements : -> -> - Exclude internal states. -> -> - Second level Job states are also mandatory. -> -> - Inside the 'Submitted' state : -> - The Execution Service may already allocate resources, -> - Rename 'Waiting' substate as 'Validated', -> - Add 'Hold' substate. -> -> Best regards. -> -> ----------------------------------------------------- -> Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS -> Bat 200 91898 ORSAY France -> Tel: +33 1 64 46 84 87 Skype: etienne.urbah -> Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -> ----------------------------------------------------- -> -> -> On Thu, 17 Dec 2009, Riedel, Morris wrote: -> > Hi team, -> > -> > here another update on one action item: -> > -> >> - A - action (Morris): copy strawman rendering doc to drafts -> > -> > Done. -> > -> > The actual draft specification (version 0.37) is now accessible in -> GridForge via: -> > Doucments - Root Folder - Drafts - Specification Direct link: -> > -> http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi- -> > wg/docman.root.drafts.specification/doc15839 -> > -> > So far, I kept a copy within the Input Document Section since I -> propose a renaming of the document within drafts to: -> > -> > "Draft PGI Specification" -> > -> > It is neither a strawman document nor a concrete rendering - it's -> just a draft specification from PGI. -> > -> > However, of course, I seek to discuss this tomorrow in the telcon -> with all of you. -> > -> > Your co-chair, -> > Morris -> > -> > --------------------------------------------------------------------- -> - -> > -------------------------- -> > --------------------------------------------------------------------- -> - -> > -------------------------- -> > Forschungszentrum Juelich GmbH -> > 52425 Juelich -> > Sitz der Gesellschaft: Juelich -> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 -> > Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe -> > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich -> > Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. -> > Sebastian M. Schmidt -> > --------------------------------------------------------------------- -> - -> > -------------------------- -> > --------------------------------------------------------------------- -> - -> > -------------------------- -_______________________________________________ -Pgi-wg mailing list -Pgi-wg@ogf.org -http://www.ogf.org/mailman/listinfo/pgi-wg
Thanks Etienne - I apply changes to the ppt. ________________________________________ Von: Etienne URBAH [urbah@lal.in2p3.fr] Gesendet: Donnerstag, 17. Dezember 2009 20:32 An: Riedel, Morris; pgi-wg@ogf.org Cc: lodygens@lal.in2p3.fr; edges-na3@mail.edges-grid.eu Betreff: Re: OGF PGI : Draft PGI Specification Morris and all, I have updated the 'Draft PGI Specification' with version 0.38 at http://forge.gridforum.org/sf/go/doc15839?nav=1 with following improvements : - Exclude internal states. - Second level Job states are also mandatory. - Inside the 'Submitted' state : - The Execution Service may already allocate resources, - Rename 'Waiting' substate as 'Validated', - Add 'Hold' substate. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Thu, 17 Dec 2009, Riedel, Morris wrote:
Hi team,
here another update on one action item:
- A - action (Morris): copy strawman rendering doc to drafts
Done.
The actual draft specification (version 0.37) is now accessible in GridForge via: Doucments - Root Folder - Drafts - Specification Direct link: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi-wg/doc...
So far, I kept a copy within the Input Document Section since I propose a renaming of the document within drafts to:
"Draft PGI Specification"
It is neither a strawman document nor a concrete rendering - it's just a draft specification from PGI.
However, of course, I seek to discuss this tomorrow in the telcon with all of you.
Your co-chair, Morris
------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
Hi PGI folks, after reading through the current draft 0.38 from http://forge.gridforum.org/sf/go/doc15839?nav=1 and listening to a presentation by Morris, I want to make a few comments. I'll try to focus on compute functionality, to keep this mail reasonably short. Overall I think the PGI looks very promising and I really appreciate your hard work! Having been present in the UMD/EMI project preparation I know exactly how hard it can be ;) So here goes. 0) the requirements doc mentioned in the introduction is not accessible for lesser mortals, on https://forge.gridforum.org/sf/go/doc15590 I get "permission denied". Maybe you could copy it to the pgi-wg area? 1) CreateActivity Since the validation steps can take some time, it is impractical to wait for these steps to finish before assigning activity IDs and returning the response. Clients or intermediaries will run into timeouts. The system should create the activities immediately, and assign them a state like "new" or "validating". IMO every remote operation that can take more than a couple of seconds to generate the response should be made asynchronous. Just think of held locks and shared resources like DB connections together with concurrent access by many clients... we've been there with UNICORE and have been forced to keep web service processing times as low as possible. 2) Change activity state I don't really see a reason for all this generic stuff. In reality you want to start, abort, hold, resume etc the processing of an activity, so why not make this more explicit. A compromise might be to do something like requestActivityStateChange("Hold"), etc, and define the mandatory list of "target states" supported by this operation. 3) Cancel activity Isn't this a special case of "Change activity state" ? 4) Wipe dito 5) Delegation port type Nice idea. However you should support also SAML assertions here (proxy certs are so 1995!) 6) in Section 5.1.2 What does "automatic resubmission" mean? Resubmission to the batch system? Or do you possibly see the PGI execution service as something "above" a normal execution service (like e.g. a glite wms?). Section 5.1.7 seems to support this view. IMO resubmitting a failed job to the batch system makes no sense, it will probably just fail again ;) So what is the idea? 7) "Delegated" state (Section 5.1.4) Allowing to delegate to an off-site execution service (like a different Grid middleware) adds complexity and messes up a lot of things, like credential delegation, state, working directory access, etc etc. Should "PGI execute" not focus on a simple, practical service for job execution? This forwarding business seems to be quite out of scope... How shall manual data staging be done if the session directory is off-site? In the intro it lists "request routing" as a requirement, but I'd reconsider that. 8) "Output sandbox" I'd try to avoid glite specific terms :) Maybe the "directory containing the output files produced by the job". At least define the term "output sandbox" somewhere. 9) I fully support Steven's statements regarding the reuse of JSDL. In some places you duplicate parts that already exist in JSDL and JSDL-POSIX, sometimes with less functionality. Some examples: - 7.2 executable name, path, arguments. This can be done by a JSDL-Posix element, which covers even more, such as environment, stdout/err/in. - 7.3.1.4 UserTag can be replaced by JSDL JobAnnotation - 7.3.6.2 Input,output,error,environment -> JSDL-Posix IMO JSDL-Posix (possibly with extensions) can be used in all places where you need to directly specify the execution of a process. Similarly the normal Application (ApplicationName, ApplicationVersion) (again possibly with extensions) can be used to define execution of a pre-installed software. 10) other JSDL related comments - 7.3.2.9 LogDir in the interest of interoperability I'd assume that the internals of how a middleware stores its "grid-specific diagnostics" is irrelevant to the job description. E.g. UNICORE would store this in a database, not in a directory on the execution system. 11) In general it is not clear to me which of these elements MUST be supported by a PGI implementation. 12) 7.3.2.14 Start time. This is reservation functionality which opens a new can of worms :-) What happens if the RMS does not support this, or the request cannot be granted? If you want to support reservation, you need to reflect this in the state model and in the possible errors a user might get. Also reservation is not listed as a requirement in the Introduction. 13) 7.3.2.15 Notifications This should not be "custom format" but "comma separated list of e-mail addresses" Summarizing: I like the port types and the basic data and execution model, also data staging and credential delegation looks good. You should re-consider the job description part and clearly identify the minimal set that has to be supported by every compliant implementation. Also I'd try to keep all implementation-specific behaviour out of the spec, like where logs are stored and what is purged by a "purge" operation. What is important is the behaviour and session directory access that a user can expect of any PGI service in each activity state (maybe a table would be helpful). Best regards, Bernd. -- Dr. Bernd Schuller Distributed Systems and Grid Computing Juelich Supercomputing Centre, http://www.fz-juelich.de/jsc Phone: +49 246161-8736 (fax -8556) Personal blog: www.jroller.com/page/gridhaus ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
Balazs, Morris, Luigi, Johannes and all, Lot of thanks to Bernd for his remarks and suggestions concerning the Draft PGI Specification. 1) CreateActivity ----------------- You can NOT have SIMULTANEOUSLY : - Vectors of Jobs, each vector containing many jobs, - Full Job Validation (including even Storage Reservation for early stage-in), - Quick response time in order to avoid timeouts when returning JobIds. Therefore : 1.1) If you need 'Vectors of Job' and 'Full Job Validation', you have to give up 'Quick response time', and manage timeouts. 1.2) If you need 'Full Job Validation' and 'Quick response time', then you have submit only one Job at a time (give up 'Vectors of Job'). 1.3) If you need 'Vectors of Job' and 'Quick response time', then you must perform only XML, Schema, and Semantic validation before returning JobIds (give up functional validation, in particular Storage Reservation for early stage-in). This requires either that : - The Job Submitter repeatedly polls the Job Status, or - We use a messaging layer permitting notifications. We have to make clear choices. 2) Change activity state ------------------------ Is Bernd taking about the 'State Model' itself ? If yes, I confirm that this model is absolutely necessary to verify that operations are consistent and can really be implemented inside an Execution Service. For an activity (= Job), we need to manage : - NOT only the Processing state (start, abort, hold, resume, ...), where the Execution Service dedicates all needed Computing resources the Job, - but also other lengthy states where the Execution Service can allocate these Computing resources to other Jobs : - Manual Stage-in (or other Pre-processing tasks), - Manual Stage-out (or other Post-processing tasks). This is clearly shown by the current 'State Model'. 7) 'Delegated' state -------------------- I strongly confirm that you can NOT suppose that the Execution Service has an intimate knowledge of the Batch System which will really execute the Job. - Execution Services are NOT obliged to implement 'Manual Stage-in'. In case of delegation to an Off-site Execution Service (like a different Grid middleware), Manual Stage-in simply MAY be impossible. - In case of delegation to an off-site execution service, the Job State is 'Delegated:Running'. An Execution Service MAY provide an EPR permitting the Job Submitter to query the precise Job State inside the Off-site Execution Service (recursively if needed). - Credential delegation is a real issue which we have to investigate. It can be performed automatically ONLY using Security profiles. By now, the EDGeS project has solved Credential Delegation in an ad hoc manner. See : 'PGI Security Model' at http://forge.gridforum.org/sf/go/doc15584?nav=1 Presentation of Security Profiles by Morris (I do NOT find it at the moment. Morris, please help.) 'Security related parts for GES strawman' at http://forge.gridforum.org/sf/go/doc15836?nav=1 +----------------+ | Other points | +----------------+ Meeting Notes ------------- Can Johannes systematically save under GridForge the meeting notes after each telephone conference ? Draft Specification Document ---------------------------- As I remember from the previous telephone conference, this document should have been : - updated with agreements achieved at this telephone conference, - renamed as 'PGI Execution Service Specification' Client-side UUID for Job identification --------------------------------------- After a little thinking : Client-side UUID can NEVER be guaranteed to be really unique. Any client could easily poison Grid databases with repeated identical UUIDs (maliciously, or by client misconfiguration, ...). So I am most strongly opposed to use Client-side UUID for Job identification. It is still possible to add in the 'PGI Execution Service Specification' that an Execution Service MAY accept (potentially non-unique) Client tags, and MAY permit the client to retrieve the list of all Jobs having a given tag. Summary of previous discussions ------------------------------- - Consistency between the 'State model' and the Operations (which are supposed to be implemented using SOAP). There are 3 ways to implement message transfers between Job Submitter and Execution Service : - Execution Service and Job Submitter both implement notifications : This permits to avoid the Job Submitter to repeatedly poll the Job status, but this is NOT easily supported by SOAP. - Job submitter has to poll the Job status. In order to minimize polling, the Execution Service MAY perhaps return the stage-in location early inside the 'Submitted' state, but this requires that the Execution Service is really able to perform stage-in inside the 'Submitted' state, which is NOT always the case. - For the 'CreateActivity' operation : - The Execution Service MAY perhaps provide the stage-in location at this early stage (this strongly impacts the 'Submitted' state), - Proposal to add a 'Notification EPR' as an additional optional input parameter. - Exclude internal states (in particular, suppress the 'Submitted:Incoming' substate). - Second level Job states are also mandatory (but third level Job states are still optional). - Inside the 'Submitted' state : - The Execution Service can return a JobId to the submitter only if it has completely validated the JSDL of the Job, therefore rename 'Waiting' substate as 'Validated', - The Execution Service MAY perhaps already allocate resources, therefore add 'Hold' substate, which could permit early stage-in. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Thu, 14/01/2010 10:59, Bernd Schuller wrote:
Hi PGI folks,
after reading through the current draft 0.38 from http://forge.gridforum.org/sf/go/doc15839?nav=1 and listening to a presentation by Morris, I want to make a few comments. I'll try to focus on compute functionality, to keep this mail reasonably short.
Overall I think the PGI looks very promising and I really appreciate your hard work! Having been present in the UMD/EMI project preparation I know exactly how hard it can be ;)
So here goes.
0) the requirements doc mentioned in the introduction is not accessible for lesser mortals, on https://forge.gridforum.org/sf/go/doc15590 I get "permission denied". Maybe you could copy it to the pgi-wg area?
1) CreateActivity Since the validation steps can take some time, it is impractical to wait for these steps to finish before assigning activity IDs and returning the response. Clients or intermediaries will run into timeouts. The system should create the activities immediately, and assign them a state like "new" or "validating". IMO every remote operation that can take more than a couple of seconds to generate the response should be made asynchronous. Just think of held locks and shared resources like DB connections together with concurrent access by many clients... we've been there with UNICORE and have been forced to keep web service processing times as low as possible.
2) Change activity state I don't really see a reason for all this generic stuff. In reality you want to start, abort, hold, resume etc the processing of an activity, so why not make this more explicit. A compromise might be to do something like requestActivityStateChange("Hold"), etc, and define the mandatory list of "target states" supported by this operation.
3) Cancel activity Isn't this a special case of "Change activity state" ?
4) Wipe dito
5) Delegation port type Nice idea. However you should support also SAML assertions here (proxy certs are so 1995!)
6) in Section 5.1.2 What does "automatic resubmission" mean? Resubmission to the batch system? Or do you possibly see the PGI execution service as something "above" a normal execution service (like e.g. a glite wms?). Section 5.1.7 seems to support this view. IMO resubmitting a failed job to the batch system makes no sense, it will probably just fail again ;) So what is the idea?
7) "Delegated" state (Section 5.1.4) Allowing to delegate to an off-site execution service (like a different Grid middleware) adds complexity and messes up a lot of things, like credential delegation, state, working directory access, etc etc. Should "PGI execute" not focus on a simple, practical service for job execution? This forwarding business seems to be quite out of scope... How shall manual data staging be done if the session directory is off-site? In the intro it lists "request routing" as a requirement, but I'd reconsider that.
8) "Output sandbox" I'd try to avoid glite specific terms :) Maybe the "directory containing the output files produced by the job". At least define the term "output sandbox" somewhere.
9) I fully support Steven's statements regarding the reuse of JSDL. In some places you duplicate parts that already exist in JSDL and JSDL-POSIX, sometimes with less functionality. Some examples:
- 7.2 executable name, path, arguments. This can be done by a JSDL-Posix element, which covers even more, such as environment, stdout/err/in.
- 7.3.1.4 UserTag can be replaced by JSDL JobAnnotation
- 7.3.6.2 Input,output,error,environment -> JSDL-Posix
IMO JSDL-Posix (possibly with extensions) can be used in all places where you need to directly specify the execution of a process. Similarly the normal Application (ApplicationName, ApplicationVersion) (again possibly with extensions) can be used to define execution of a pre-installed software.
10) other JSDL related comments - 7.3.2.9 LogDir in the interest of interoperability I'd assume that the internals of how a middleware stores its "grid-specific diagnostics" is irrelevant to the job description. E.g. UNICORE would store this in a database, not in a directory on the execution system.
11) In general it is not clear to me which of these elements MUST be supported by a PGI implementation.
12) 7.3.2.14 Start time. This is reservation functionality which opens a new can of worms :-) What happens if the RMS does not support this, or the request cannot be granted? If you want to support reservation, you need to reflect this in the state model and in the possible errors a user might get. Also reservation is not listed as a requirement in the Introduction.
13) 7.3.2.15 Notifications This should not be "custom format" but "comma separated list of e-mail addresses"
Summarizing: I like the port types and the basic data and execution model, also data staging and credential delegation looks good. You should re-consider the job description part and clearly identify the minimal set that has to be supported by every compliant implementation. Also I'd try to keep all implementation-specific behaviour out of the spec, like where logs are stored and what is purged by a "purge" operation. What is important is the behaviour and session directory access that a user can expect of any PGI service in each activity state (maybe a table would be helpful).
Best regards, Bernd.
-- Dr. Bernd Schuller Distributed Systems and Grid Computing Juelich Supercomputing Centre, http://www.fz-juelich.de/jsc Phone: +49 246161-8736 (fax -8556) Personal blog: www.jroller.com/page/gridhaus
------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ _______________________________________________ Pgi-wg mailing list Pgi-wg@ogf.org http://www.ogf.org/mailman/listinfo/pgi-wg
Hi Bernd, All, thanks for your valuable comments. I put them in the PGI working slides to be more efficiently discussed and tracked. When there is time enough these points will be discussed in the telcon today. Take care, Morris ------------------------------------------------------------ Morris Riedel Jülich Supercomputing Centre (JSC) Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel "We work to better ourselves, and the rest of humanity" Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender)
------Original Message----- -From: pgi-wg-bounces@ogf.org [mailto:pgi-wg-bounces@ogf.org] On Behalf Of Bernd Schuller -Sent: Thursday, January 14, 2010 10:59 AM -To: pgi-wg@ogf.org -Cc: UNICORE devel -Subject: Re: [Pgi-wg] OGF PGI : Draft PGI Specification - -Hi PGI folks, - -after reading through the current draft 0.38 from -http://forge.gridforum.org/sf/go/doc15839?nav=1 and listening to a -presentation by Morris, I want to make a few comments. -I'll try to focus on compute functionality, to keep this mail reasonably -short. - -Overall I think the PGI looks very promising and I really appreciate -your hard work! Having been present in the UMD/EMI project preparation I -know exactly how hard it can be ;) - -So here goes. - -0) the requirements doc mentioned in the introduction is not accessible -for lesser mortals, on https://forge.gridforum.org/sf/go/doc15590 -I get "permission denied". Maybe you could copy it to the pgi-wg area? - -1) CreateActivity -Since the validation steps can take some time, it is impractical to wait -for these steps to finish before assigning activity IDs and returning -the response. Clients or intermediaries will run into timeouts. The -system should create the activities immediately, and assign them a state -like "new" or "validating". IMO every remote operation that can take -more than a couple of seconds to generate the response should be made -asynchronous. Just think of held locks and shared resources like DB -connections together with concurrent access by many clients... we've -been there with UNICORE and have been forced to keep web service -processing times as low as possible. - -2) Change activity state -I don't really see a reason for all this generic stuff. In reality you -want to start, abort, hold, resume etc the processing of an activity, so -why not make this more explicit. A compromise might be to do -something like requestActivityStateChange("Hold"), etc, and define the -mandatory list of "target states" supported by this operation. - -3) Cancel activity -Isn't this a special case of "Change activity state" ? - -4) Wipe -dito - -5) Delegation port type -Nice idea. However you should support also SAML assertions here (proxy -certs are so 1995!) - -6) in Section 5.1.2 -What does "automatic resubmission" mean? Resubmission to the batch -system? Or do you possibly see the PGI execution service as something -"above" a normal execution service (like e.g. a glite wms?). Section -5.1.7 seems to support this view. -IMO resubmitting a failed job to the batch system makes no sense, it -will probably just fail again ;) So what is the idea? - -7) "Delegated" state (Section 5.1.4) -Allowing to delegate to an off-site execution service (like a different -Grid middleware) adds complexity and messes up a lot of things, like -credential delegation, state, working directory access, etc etc. -Should "PGI execute" not focus on a simple, practical service for job -execution? This forwarding business seems to be quite out of scope... -How shall manual data staging be done if the session directory is -off-site? -In the intro it lists "request routing" as a requirement, but I'd -reconsider that. - - -8) "Output sandbox" -I'd try to avoid glite specific terms :) Maybe the "directory containing -the output files produced by the job". At least define the term "output -sandbox" somewhere. - -9) I fully support Steven's statements regarding the reuse of JSDL. In -some places you duplicate parts that already exist in JSDL and -JSDL-POSIX, sometimes with less functionality. Some examples: - - - 7.2 executable name, path, arguments. This can be done by a -JSDL-Posix element, which covers even more, such as environment, -stdout/err/in. - - - 7.3.1.4 UserTag can be replaced by JSDL JobAnnotation - - - 7.3.6.2 Input,output,error,environment -> JSDL-Posix - -IMO JSDL-Posix (possibly with extensions) can be used in all places -where you need to directly specify the execution of a process. Similarly -the normal Application (ApplicationName, ApplicationVersion) (again -possibly with extensions) can be used to define execution of a -pre-installed software. - -10) other JSDL related comments - - 7.3.2.9 LogDir in the interest of interoperability I'd assume that -the internals of how a middleware stores its "grid-specific diagnostics" -is irrelevant to the job description. E.g. UNICORE would store this in a -database, not in a directory on the execution system. - -11) In general it is not clear to me which of these elements MUST be -supported by a PGI implementation. - -12) 7.3.2.14 Start time. This is reservation functionality which opens a -new can of worms :-) What happens if the RMS does not support this, or -the request cannot be granted? If you want to support reservation, you -need to reflect this in the state model and in the possible errors a -user might get. Also reservation is not listed as a requirement in the -Introduction. - -13) 7.3.2.15 Notifications This should not be "custom format" but "comma -separated list of e-mail addresses" - - -Summarizing: I like the port types and the basic data and execution -model, also data staging and credential delegation looks good. You -should re-consider the job description part and clearly identify the -minimal set that has to be supported by every compliant implementation. -Also I'd try to keep all implementation-specific behaviour out of the -spec, like where logs are stored and what is purged by a "purge" -operation. What is important is the behaviour and session directory -access that a user can expect of any PGI service in each activity state -(maybe a table would be helpful). - -Best regards, -Bernd. - - --- -Dr. Bernd Schuller -Distributed Systems and Grid Computing -Juelich Supercomputing Centre, http://www.fz-juelich.de/jsc -Phone: +49 246161-8736 (fax -8556) -Personal blog: www.jroller.com/page/gridhaus - - ------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------- -Forschungszentrum Juelich GmbH -52425 Juelich -Sitz der Gesellschaft: Juelich -Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 -Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe -Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), -Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, -Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------- -_______________________________________________ -Pgi-wg mailing list -Pgi-wg@ogf.org -http://www.ogf.org/mailman/listinfo/pgi-wg
participants (5)
-
Bernd Schuller -
Etienne URBAH -
Morris Riedel -
Riedel, Morris
-
Steven Newhouse