- pgi-wg - lists.ogf.org

Draft Specification Discussion Slides
by Morris Riedel 14 Jan '10

14 Jan '10

Hi team, as the most of you remember, I used during the last telcon animations within a ppt in order to understand our challenges in more detail. I put the ppt into GridForge including our last known discussion problem: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.pgi-wg/do… Take care, Morris ------------------------------------------------------------ Morris Riedel Jülich Supercomputing Centre (JSC) Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel "We work to better ourselves, and the rest of humanity" Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender)

1 0

FW: PGI Telcon on Friday 2010-01-15; 4 pm CET
by Morris Riedel 14 Jan '10

14 Jan '10

Send once again since it seems to be blocked somewhere... -----Original Message----- From: Morris Riedel [mailto:m.riedel@fz-juelich.de] Sent: Thursday, January 14, 2010 2:11 PM To: 'Pgi-wg(a)ogf.org' Cc: 'gin(a)ogf.org' Subject: PGI Telcon on Friday 2010-01-15; 4 pm CET Hi team, first of all: Happy new year and thanks for your contributions to the group in 2009. As agreed in the last telcon we will have our first 2010 telcon tomorrow 2010-01-15 at 4 p.m (CET): (1) Numbers: To join the PGI teleconference, please use this calling information: via Skype call +9900827049931906 (free of charge), ordinary phone numbers (local rates) with the 9931906 conference number: Austria 0820 401 15470 Belgium 0703 57 134 France 0826 109 071 Germany +49 (0) 180 500 9527 Switzerland 0848 560 397 (2) Screen-Sharing: http://webconf.vc.dfn.de/ogf-pgi-wg/ No login is required. Just enter as 'guest' with your real world name. (3) Proposed agenda: (i) Debrief minutes of last PGI TelCon (ii) Debrief PGI actions (iii) Discuss regular change of telcon timeslot in 2010 instead of Friday afternoon (iv) Continue discussion animation ppt of PGI operation & state model Focus: Waiting for a job-id until all validation steps have been performed is unrealistic from implementation view (e.g. 100 jobs in submit array) (v) Debrief Steven Newhouse comments on the list (if time) (vi) Debrief Bernd Schuller comments on the list (if time) (vii) Interop demonstration Setups for OGF28 (viii) Next meeting (ix) AOB Please feel free to add other items on the agenda as you like. A high participation from the PGI and GIN members would be appreciated. Finally, I will upload the animation ppt and the last up to date document today. Your co-chair, Morris ------------------------------------------------------------ Morris Riedel Jülich Supercomputing Centre (JSC) More: http://www.fz-juelich.de/jsc/JSCPeople/riedel "We work to better ourselves, and the rest of humanity" Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender)

1 0

Merry Christmas
by Johannes Watzl 24 Dec '09

24 Dec '09

Dear PGI members, I wish you a merry Christmas and a happy new year! All the best for you and PGI in 2010! Best, Johannes -- -- _ _ _ _ _ _ Johannes Watzl |\/| |\ | |\/| Institut für Informatik / Dept. of CS | | | \| | | Ludwig-Maximilians-Universität München ======= TEAM ======= Oettingenstr. 67, 80538 Munich, Germany Room E 005, Phone +49-89-2180-9162 Munich Network Management Team Email: watzl(a)nm.ifi.lmu.de Münchner Netz-Management Team http://www.nm.ifi.lmu.de/~watzl

1 0

OGF PGI - UML Sequence Diagram as Step by step base like figure of presentation
by Etienne URBAH 18 Dec '09

18 Dec '09

Morris and all, At the last OGF PGI telephone conference, Moris had asked for a 'Step by step base like figure of presentation'. With ArgoUML, I have created the attached UML Sequence Diagram. Does such an UML Sequence Diagram suit your needs ? I am open for all remarks and suggestions. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------

2 1

PGI Telcon on Friday 2009-12-18 Screensharing Link & Test after 3 pm
by Riedel, Morris 18 Dec '09

18 Dec '09

Hi folks, I setup a meeting space link that is as follows: http://webconf.vc.dfn.de/ogf-pgi-wg/ Please use this link and perform the following steps: (1) Enter as a guest, please use a name that clearly identifies you (2) Hit"Enter Room" I will be in this screen sharing environment already one hour before the meeting starts (3 p.m. CET) to give you all the opportunity to test the system before we start at 4 p.m. CET. I'm looking forward to better progress now, but keep in mind that I also use this system for the first time in "production" - but early tests have been great. Take care, Morris ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Update on PGI Actions (gLite/CREAM)
by Riedel, Morris 17 Dec '09

17 Dec '09

Hi folks, update on two actions in one: >- A - action (Morris): should arrange that someone from gLite is there >- A - action (Morris, all): strengthen gLite participation I have asked Luigi if he joins tomorrow and also put this topic on the agenda for tomorrow to discuss how we can improve in this context - maybe we should again shift our PGI telcons to another more attractive spot than Friday afternoon. Take care, Morris ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

2 2

Update on PGI Action (PGI requests for OGF28)
by Riedel, Morris 17 Dec '09

17 Dec '09

Hi folks, here a little update on the following task: >- A - action (Morris): session proposals -> call for sessions As we discussed the 3 PGI working sessions should be part of the interoperability day that was already agreed with the local chair man (Nils) and therefore I requested no official sessions yet. However, I contacted Nils to clarify things. Take care, Morris ________________________________________ Von: Riedel, Morris Gesendet: Donnerstag, 17. Dezember 2009 09:23 An: pgi-wg(a)ogf.org Betreff: Update on PGI Action (BES @ OGF28) Hi team, a short update on the following PGI Action: >- A - action (Morris): check if there is a BES session, discuss with Andrew I asked the BES chairs Andrew and Steven via E-Mail and proposed at least one joint PGI/BES Session since their is no BES session so far in the current requested sessions list for OGF28. Your co-chair, Morris ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Update on PGI Action (BES @ OGF28)
by Riedel, Morris 17 Dec '09

17 Dec '09

Hi team, a short update on the following PGI Action: >- A - action (Morris): check if there is a BES session, discuss with Andrew I asked the BES chairs Andrew and Steven via E-Mail and proposed at least one joint PGI/BES Session since their is no BES session so far in the current requested sessions list for OGF28. Your co-chair, Morris ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Next PGI meeting - Tuesday, 8 Dec 2009, 16:00 (CET)
by Johannes Watzl 11 Dec '09

11 Dec '09

Dear all, just to remind. As agreed on Friday, our next meeting will be held tomorrow. Tuesday, 8 Dec 2009, 16:00 (CET)! Best, Johannes -- _ _ _ _ _ _ Johannes Watzl |\/| |\ | |\/| Institut für Informatik / Dept. of CS | | | \| | | Ludwig-Maximilians-Universität München ======= TEAM ======= Oettingenstr. 67, 80538 Munich, Germany Room E 005, Phone +49-89-2180-9162 Munich Network Management Team Email: watzl(a)nm.ifi.lmu.de Münchner Netz-Management Team http://www.nm.ifi.lmu.de/~watzl

3 2

Meeting - Tuesday, 8 Dec 2009, 16:00 (CET) - Notes
by Johannes Watzl 08 Dec '09

08 Dec '09

Dear all, please see below the notes from today's meeting. Best, Johannes Meeting - Tuesday, 8 Dec 2009, 16:00 (CET) Participants: Morris, Balazs, Etienne, Johannes, Aleksander Agenda: Discussion of the work of Etienne on the "OGF PGI - AGU Execution Service Strawman Rendering" Morris: go through meeting notes from last meeting would be good to have gLite participation Balazs: we need at least one more person from gLite real work can only be done with a screensharing tool just discuss the ideas behind Etienne's work Balazs (chairing the strawman rendering document discussion): doc is the latest version of the doc (version 37) by Aleksander on the PGI GridForge page page 4 synchronize, merge Etiennes's cahnges with actual model in createActivity (second sentence) Etienne: the initial state of a job is the submitted state Balazs: substate? Etienne: incoming, waiting, outgoing input queue, has not been processed yet submitted waiting: it is being processed submitted outgoing: id or epr has been given to the job, ready for next state change that in a consistent way Balazs: combining substates in a certain way -> clearify more not combine submitted outgoing with response Etienne: useful that each state will represent something if you do not give job id at the end of submitted state, you have to completely include the submitted state Balazs: only the first level states, 2nd third are optional? Etienne: important: at the end of submitted state - job id associated Balazs: why at the end? Etienne: at least at the end! could be returned earlier also it is useful for the execution service for checking if it is possible (JSDL checking) Balazs: not validate JSDL here when execution service returns response? Morris: page 4, comments in terms of UNICORE it is alway possible directly, you don't do brokering not exactly sure how this relates to the state model out of the scope of the session Balazs: not discussing staging of data discussing in which state job should be does not see this written down now Morris: it is in the job description document validation Balazs: missing the satet model how do you read the job description validation? Morris: matchmaking if there is a must - there is a must in gLite, first from the broker, then from execution service Balazs: first mandatory validation, then respose, maybe validation errors you have to validate first, to check for errors Etienne: first mandatory validations: XML schema, semantic some validation may be only possible later?! Balazs: all validation should be done before then assign state you submit a job, execution service does couple of things like validation the first time you can check the state, when you have the job id Etienne: in the beginning did not write the internal second level submit state added them for some request ready to remove them Balazs: hold point is important substate of the submitted which is a hold! Etienne: first sent on 13 Aug Balazs: changeState operation resume like operations andy kind of hold states Morris: need a figure not the state model figure should have a step by step base like figure of presentation Etienne: agree of the scope of the submitted state inside the submitted state as soon has a job id, the job should go to the preprocessing state, where it can be held submitted state a certain time, in the beginning already job id can go to a hold substate Balazs: preprocessing: different data staging Etienne: preprocessing includes data staging any other actions performed that's why third level hold we have another preprocessiong hold suspended when user wants to hold withoud relationship to staging Morris: i.e. preprocessiong stage, only one CPU before using the whole machine not related to data staging Balazs: proprocessing is very complex state Morris: during the execution vs. pre and post processing "before finished" other requirement for putting it somewhere Etienne: postprocessing hold suspened state Morris: clearly identify it is postprocessing - job is finished Etienne: can go in any order at any time you want Balazs: pre, post processing very complex, dangerous does not see a sequence how the job goes through Balazs: really would like to see a linear state model Etienne: agree with Balazs, inside the preprocessing hold there is no clear idication in which order the third level substates are processed implicitely suppose: can be processed in any order and any number of times achieved only ba the implementation then it is ok the order must be well known from the beginnging, we have to renew the state model Balazs: when exactly this will happen? put the job in a hold before any kind of data staging takes place Etienne. if we keep model exacly as it is, it is not very precise, implementations can do it more easy if we create a too precise model, it will be only implemented by one middleware to many details, will limit the chance to have different middlewares are you shure you want to do that? Balazs: just want to stop the job before preprocessing Morris: use case? you don't do data staging then? Balazs: just to be sure, the job does no other things for safety reason Morris: pre means before preprocessing is not data staging Balazs: in ARC it is a safety measure to be able to stop the job before running Morris: does not really see the difference user gets the feedback -> submitted submitted hold sounds strange no destinction to preprocessing Balazs: submitted hold is the end of submitted Morris. seeking to understand the difference consistency in the sate model automatic resubmission people may want to specify certain points use case maybe accounting problems Etienne: inside the submitted state, reformulate substates one where the jobid is associated to the job after second job already has the job id, user is able to perform an operation on the job the jobid is associated in the middle of the submitted state there can be a submitted hold where the id has already given to the job we have to synchronize the state model with the messages exchanged Aleksander: how can it be an activity without an id Morris: validation steps createActivity operations should only be successful if validation successful if validation not successful, no job created, no hold Etienne: in figure rename submitteds waiting to submitted validating add another substate, submitted hold id must be associated to the job corresponding messages have to be sent back to the user validation, creation of id, then send id to user Morris: put still the waiting Etienne: rename waiting by validating Balazs: everything is done, job is ready to go to the next state preprocessing, delegating, postprocessing Etienne: in which ord third level substates are processed? Balozs: hold blocks blocking before going to the next state Etienne: second level hold substate before outgoing substate? Balazs: yes Etienne: understands the requirement infact in the three secondlevel states preproc,delegating postproc secondlevel hold substate corresponds to what Balazs wishes Balazs: different states secondlevel hold, thrid level hold? Morris: there might be middlewares without third level Balazs: needs preprocessing hold Morris: how is the third level flow? Balazs: optional: third, second level Morris: disagree second level should be adopted from all submitted, preprocessing, delegating, postprocessing i.e. preprocessing hold failed recoverable not many middlewares can do this agree with Balazs: aligning the thrird level of states Etienne: first two levels are mandatory inside submitted: add a hold substate Morris: submitted incoming is missing comparing to the state model submitted incoming is a state or is it no state? Balazs: job ide already Etienne: no initial substate of the submitted state state without a name remove submitted incoming at this point no job id yet - cannot be handled just internal state Morris: problems reported in logfiles Etienne: expose only states which can be used by the submitter remove the submitted incoming label - it is not relevant to the submitter agreement on what is going on in the submitted state. Morris: submitted validated only job id if it is validated Etienne. job id is given at validated substate return to the document inside the submitted state, the execution servcie cannot always provide the location Balazs: switch to the data staging issue data staging on createActivity pull approach service can publish information Etienne: mail: three ways for message transfer which way is the best? Balazs: client needs the info about the data staging location Etienne: have to repeatedly poll until the job comes to manual staging if there is manual staging the job goes the preprocessing state, hold cannot guarantee the consistency of that - to be discussed perhaps not so easy to link the return to a late state Morris. one figure not enough maybe an animation hopefully can work on this ugently needed here!!! Balazs: strongly avoid notification publish the staging location, not be coupled to job state server needs time to figure out the staging location - publish after some time why should this be coupled to a job state?? Etienne: in some cases the service has inner knowledge about the storage location inside the submitted state Balazs: job should not be preprocessing before it does not know the staging location Morris: compared to job description validation i.e. we need 2 GB, requirement how it could be still vaild to have it in submitted? Etienne: computing element whcih really owns storage but if it is divided into sequential parts where submitted parts do not know about physical storage but of batch systems or strage systeme, there is a problem Etienne: at validated an execution service may already have allocated storage Balazs: we should edit the text when everybody can see the document Aleksander: validation, allocation data staging after allocation id should be assigned as soon job is validated client receives information, polling or notification Morris: continue next time with desktop sharing tool Etienne: are these three ways from email consistent? which ways are preferred? Aleksander: resend the mail! Etienne: sent on 26 Oct, 4 Dec Morris: continue next week -- _ _ _ _ _ _ Johannes Watzl |\/| |\ | |\/| Institut für Informatik / Dept. of CS | | | \| | | Ludwig-Maximilians-Universität München ======= TEAM ======= Oettingenstr. 67, 80538 Munich, Germany Room E 005, Phone +49-89-2180-9162 Munich Network Management Team Email: watzl(a)nm.ifi.lmu.de Münchner Netz-Management Team http://www.nm.ifi.lmu.de/~watzl

2 1