PGI f2f Workshop Amsterdam, 29-30 April 2010 Meeting notes participants in Amsterdam: Morris, Shabhaz, Balazs, Etienne, Luigi, Johannes on the call: Steve Crouch, Tim Parkinson, Andrew, Mark, Emmanouil, Oxana, David Wallom, David Snelling Start: 9:55 list in wiki with ~210 requirements condensed list created over the last days in gridForge dropped items marked as "dropped" in details questions? Andrew: should use "raise hand" functionality Andrew: any correlation between numbers/ids of old and new version Etienne: numbers of spreadsheet Morris: screensharing - raise hand start at the beginning of the list Morris: look at the process review David S: progress on refining requirements next steps, outcome of this workshop: agreement mutual understanding on requirments Andrew: some requirmenents got dropped, want to bring some of them back Morris: outcome: agreed list of requirements in scope of PGI Balazs: if we want to make decisions - want to see the process of decision making Morris: should talk about David S: recommend to quickly go through the requirements general agreement Balazs: one objection is enough to do it in the second round Morris: comments to the process? Dave S: everybody has to understand it first and then quickly accept or reject it once you have a requirement what to do next what is the next step in the process? Morris: everybody can come up with a strawman Balasz: forward the list to BES, JSDL,... groups David S: call for suggestions? Andrew: need some criteria to select David S: how long do people have to put together strawman proposals Johannes: collect by OGF29 select at OGF30 Morris: problems with this timeline? David W: should work on strawmen at OGF29 Morris: don't have to be completed totally but in a shape to be able to discuss them Andrew: more important to get good starter strawmen from different directions Dave S: by end of OGF29 you need to have chosen one Morris: would be tough David: go through the process on the screensharing Morris: (1) yes - no triage quick list of condensed list pdf - re-proposed ha everybody agrees to say yes and keep them quickly (2) when we have the requirements ... any strawman that answers as many requirements as possible might be numerous ones (3) Strawman collected At OGF29 jUne 20th Decided on (4) Select strawman by OGF30 David S: hos does this fit with other pojects? Morris: strawman from european perspective October is a little bit late but ok David S: strawman decided by beginning of August make a decision before OGF30 Morris: mabe no collection at OGF29 another f2f in August to support the decision process to be able to present the decision at OGF30 what about others in the group? US? Andrew: busy til mid of July continue the work in GIN timeline works vacation time may be a problem Morris: then just two weeks to decide for you OGF29 for discussions of strawmen drafts do you think two weeks are enough? Andrew: negotiation would take more than two weeks people can come to positions in two weeks Morris: aim: save time Balazs: no decision can be made before august, unrealistic to have a decision before august Morris: end of August deadline? David W: indroducing delays global effort! keep 31 July David S: if it is important enough, people will increase the pace Morris: increase the pace now or accept the delays David S: decision mid of July solve problems with european holidays Morris: problem in the US Andrew: we will find a way Morris: instead of 31 August -> mid of July Luigi: EC has an explicit requirement we need an agreement, specification, cannot wait too much time we need it before August project starts 1 May Morris: Luigi pointed out even mid of July too late Luigi: we have to finalize the list and then go on Balazs: commitment need 2 months of dedicated work for PGI Dave S: decision as early as possible! mid of July seems possible right track push mid July decision focus on requirements Balazs: dedicated effort needed Luigi: roadmap: we will have strawman Morris: how to note down the process? proposal: Etienne note down "yes" or "no" Etienne: would like to trace the information in the wiki Balazs: desaster, too slow! google document David W: in this session go through requirments which were dropped if anybody has concerns or wants discuss dropped requ. in the next session go through two groups on the call who have a number of dropped requirements they want to have discussed Morris: we will lose a lot of time follow Dave S. suggestion to go quickly through the list David W: ok Balazs: take google doc and use DFN screensharing David W: so let's start Morris: Everybody ready? Andrew: yes David: yes Emannouil: yes starting with going through the list. Morris: when we are done with quickly going through, we do the second run mark every requirement with yes, no, or unclear Requirement 13,14,15,16 everybody agrees on message level 17: no 18: same as 12 19: David W: definition of good practice, not really a requirement Andrew: maybe no authorization is out of scope Etienne: in scope of GLUE model when client submits job it is accepted or not by policy 24: David W: restriction on must? Andrew: if you are going to require you should take one at least at some point in security looking at production grids maybe this is a no 25: Oxana: mutual exclusive with 22 Andrew: 25 is too strong - should say no to 25 26: Luigi: no Andrew: for the US a hard requirement Shibboleth may be the answer Etienne: understand from user point of view but not from service point of view 27: Andrew: against, because it is telling how to implement things Etienne: it is a no 28: Andrew: method interpreted as mechanism? -> yes Oxana: what is the use of this requirement 29: David W: reject 32: must? Andrew: out of scope Oxana: authorized means authorization is granted 37: Etienne: "compute" is useless here Andrew: strong requirement from user community to have logs independant of accounting and security 41: Andrew: should be a no not a requirement 45: Andrew: unclear? 48: Andrew: no David: when it is moved it becomes new activity ---- going through rejected Andrew: NF13 as new requirement 160 also requirement from Etienne and Luigi Etienne has a list with reproposed requirements everybody else agrees with the rejected requirements. from Etienne: new requirement 161 NF6 - 162 David W: specific IS2 - 163 IS8 - 164 Steve: what is the difference between this and IS3 IS9 - 165 IS13 - 166 AC0 - 167 Andrew: non scalable requirement added 168, 169, 170, 171 setting up dimdim screensharing try first 5 unclear ones going through unclear requirements 3 Etienne: vector limits we must have one separate endpoint for each group of operations which have the same vector limits Andrew: we haven't defined what an endpoint is why do we have to split them to different endpoints change text Balazs: in the inhomogenous case more endpoints Andrew: if we send JSDL document, leave it to BES to handle it David W: maybe will become impossible to manage change to *no* ### 5: change text Balazs: in GLUE a model cannot do anything with storage Etienne: there is a global pointer to storage the execution service must only provide details about computing entities and no details about storage entities change to *yes* ### 8: Morris: scalability issue Etienne: better description form the wiki changed to *yes* ### 11: rewritten but after timeout still unclear ### 13: Mark: don't seem like a requirement Morris: they are options Luigi: there is written "may" Morris: looking at 13-16, they are all the same thing Balazs: suggestion to have a dedicated security session we need to have a global agreement of authentication/authorization in PGI Morris: we need a specific security working task ### 18: Andrew: believe should not be a requirement Morris: example? Tim: many institutions will not allow non institution members to access Etienne: if there is an anonymous request: no authorization Andrew: by saying it must require security credentials will allow anybody to use a machine certain policies will not be allowed clear, changed to *no* ### 19: Etienne: for authorization someone can send package I am manager of certain VO user in other VO and has special account on a certain Grid A man in the middle should not be able to get one of these David: about how the authorization is actually handled Etienne: you can put several SAML assertion into one single header Andrew: with SAML all you know is some attributes from somebody you trust different words to describe it Oxana: how can this prevented by packaging Etienne: solution: global signature Morris: we all should design it a way that we all can say yes changed to *yes* ### 26: Andrew: requirement from funding agency we need to support delegation and trust federation Morris: single sign-on Etienne: not understanding the last sentence Andrew: incommon is one of the providers in the united states for research and government agreement on SAML document Morris: Move to Shibboleth in Europe too change to *no* ### 29: change to *yes* ### 31: Luigi: whenever a user may need to manage a lifecycle of an activity when certificate expired activity is terminated Balazs: on the service you have credentials somewhere Luigi: you associate activity with credentials rephrasing change to *yes* ### 32: Luigi: we need a mechanism to authorize a user to manage an activity from another user Etienne: better explanation on the wiki Andrew: just add "must exist a mechanism" concerned with focus on GLUE entities information model does not concretely say anything change to *yes* close of day one. participants online: Tim, Steve, Mark Andrew, Emannouil, Aleksandr, Oxana, David participants amsterdam: Balazs, Shabhaz, Morris, Johannes, Luigi, Etienne go through unclear requirements 10 min per requirement stop 5pm 34: Andrew: 34 and 35 coupled Etienne: details on wiki page Andrew: take them out Steve: can run a grid without this Etienne: in desktop grids is mandatory Balazs: JSDL hook assume the execution service is able to deal with it Morris: maybe have it without allication repositories changed text change to *yes* ### 35: Andrew: merge 34 and 35 Etienne: global namespace separate Morris, Balazs: example change to *yes* and duplicate to 34 ### 36: Andrew: TeraGrid uses UR for accounting is it is necessary to extend UR to network Etienne: compare to 167 Morris: but we understand it? Balazs: profiling on usage record changing text Etienne: accounting records for activities change to *yes* ### 37: Andrew: sequence of things that happend to job, tracking Luigi: maybe more generic Etienne: local system logs Balazs: activity level? what do you want to log? Etienne: changes of job states Andrew: fault codes Morris: security audits, user job log change to *yes* ### 38: Etienne: proposal to mark as out of scope changing text Luigi: request operation exposing logs on the portType level Andrew: some service change to *no* ### 39: duplicate of 38 David W: change the area to accounting and logging Etienne: clearly not compute and not accounting change to *yes* ### 40: Etienne: change to accounting and logging Balazs: why different to 39? Etienene: here: should support complex query languages Balazs: different "query" in 39 and 40 change text at least one complex query language change to *no* ### 48: Andrew: you need to be able to access the information; done over the execution service? Balazs: also managing the job? Andrew: yes change to *no* ### 49: Andrew: endpoint URI or epr why 49 is a requirement Etienne: the way how clients submit request is part of the interface still unclear after 10 min discussion an illustration will help ### 50: Morris: what is activity id? different understanding Andrew: suppose activity id is an epr should be able to extract the address field from it Aleksandr: you cannot contact epr, only the service represented by epr change text David W: what is the reason for this requirement? makes it a lot harder for the group Luigi: consider activity id as an epr - flexible structure Oxana: more than one requirement in this text checking wiki page with original text still unclear after 10 min ### 51: change text Andrew: if we are operating in a WS world and strictly going with opaque ids; you need a name service to bind noticeable slower Oxana: last sencence must be removed: it is exactly the same as in requ 50 Andrew: we understand the requirements, discuss later Etienne: how can any client contact any server Morris: clarify this but not now, maybe with illustration change 50 and 51 to *no* ### 52: change to *no* ### 56: Andrew: clear to me, but reword it one way to achieve are vector operations will vote no change to *no* ### 57: Morris: bulk operation limit change to *no* ### 58: Balazs: clients should not assume that esxecution service comes with this frunctionality Morris: we have a requirement that defines something out of scope change text Mark: pretty much everybody should vote no either in or out of scope rephrase the requirement change to *no* ### 62: change to *yes* duplicate of 37 ### 63: change to *no* ### 64: change to *no* ### 69: Mark: vote no, activity ids not defined Balazs: we don't know details of activity id - general object Etienne: session directory embedded in activity id? change to *no* ### 72: Mark: difference between state and statuas information? Etienne, Balazs: here the same Etienne: replace multiple by extended Balazs: different schemas for backwards compatibility change text Etienne: Must support different models simultanously Johannes: you can just have the PGI model Morris, Balazs: yes, so MAY Mark: status information about activities change to *no* ### 73: David W: GLUE model instead of GLUE2 Morris, Etienne: wiki: LB6 example there Shabhaz: here query language, not in BES Morris: BES little bit extended with filters change to *no* ### 78: Morris: lot of things related to this David: pending state it's an optional substate? should be Balazs: pending state coupled to the batch system Etienne: accepted by the execution service Luigi: delegated state mapping between pending and delegated Morris: the mechanism must be there change to *no* ### 84: Luigi: want to do management change to *no* ### 85: change to *no* ### 87: Morris: lease submitted it, never herad of it, do it once? Luigi: you create the lease instance, negotiate the time to live define maximum lease time it is a timer David: don't understand Luigi, Balazs: avoid zombie jobs Morris: can't get rid of a job Balazs: cleanup over the zombie hjobs Luigi: network prroblems David: simple redescription Morris: rephrase final state to any state David: clearer but a no change to *no* ### 88, 89: postponed ### 90: Tim: "will be" is this expressing a requirement Balazs: will define a set of validation steps you can postpone this validation Morris: manual data staging kick off requirement complete knowledge if validated and executed Etienne: replace while by but Balazs: validation can be very heavy and time consuming some of these steps can be done later incomplete validation in the beginning and do the other steps later Etienne: execution should perform validation David: intensly complicated change to *no* ### 91: Balazs: include full validation activity creation change to *no* ### 92: Morris: somebody who cannot understand this? David: clear change to *no* ### 94: Tim: purge works only on final state Oxana: pending not a final state Tim: so you have to kill it Oxana: what is the definition of purge Morris: remove activity completely change text change to *yes* ### 96: Mark: activity gets migrated to another executionservice, the new one becomse the manager and the old one can forget about it use excution service and not BES Morris: example Mark: so ok support migration for efficiency reasons the original BES must not be longer responsible for it change to *no* ### 97: Mark: wait for Andrew postponed ### 99: Mark: execution service start and stop accepting requests Etienne: opposite of 84 related change to *no* ### 102: Etienne: automatic resubmission what is not clear here? Balazs: which layer? retry Etienne: clear or not? change to *yes* ### 104: David: should be a yes but rework the language of the requirement Oxana: job description should allow for alternative data sources David: an input task may neet data to be staged from different stages Morris: JSDL does all or nothing change text Shabhaz: no because of multiple sources for one of the data sources change to *no* ### 106: Balazs: one data file you want to stage, more sources for this file David: you have to take the input from other people Etienne: alternate sources for the same file Balazs: text ok? Oxana: use case created some data file upload it target separate target maybe more edifferent copies please copy to at least two should be able to specify in job description how many are needed Morris: data staging with and/or still unclear after 30 min ### 105: Etienne: no because of no manual data staging change to *no* ### 110: Dave: no, becasue flexibility ranges give change to *no* ### 112: Oxana: unclear because you cannot "describe a description" Morris: change description to resource Balazs: in the draft specification there are more examples Morris: believe it is well understood David: yes, but a no already in several places Balazs: job description is expressing what you want resource requirement and not application requirement clearly separation a mess in current JSDL Oxana: memory consumption can also come from other not BES related jobs Balazs: no agreement Oxana: consider virtual machine change to *no* ### 113: Balazs: structure already recommended in the draft specification Morris: again separation path as one element and the executable as the other element David: why? still unclear after 10 min ### 114: David: should be yes clear Oxana: "favorite" requirement change to *yes* ### 115: David: Oxana and myself understand but vote for no Balazs: why David: badly written Aleksandr: please propose your description David: include scale time required Oxana: new structure for job duration in units of benchmark Balazs: time requirement no benchmark requirement Etienne: if you cannot express time in seconds no time Balazs: structure containing time and benchmark composed name, value, time David: says which benchmarks? Balazs: not hardcoded benchmarks Oxana: somebody else has to scale time Balazs: you specify time as well NO agreement change to *no* 116: Etienne: totally unclear Balazs: in current JSDL data staging element not defined if it is a directory or a file idea is to identify if it is directory or file David: just copy data elemet change the wording change to *yes* ### 117: David: clear but no Balazs: for anybody unclear? ### about 30 "unclear" about 80 "no" Morris: 1st step: go through the unclears Balazs: at least two meetings for unclears Morris: looking on NOs prioritisation approach downgrade these to 30 or 40 which are very important Mark: some of the nos multiple telecons Morris: what do you think of prioritisation David: you just need to have it clearly stated how this should work David: two parallel groups to start with would be a good thing and double the speed Balazs: next thing voting - decision making Oxana: if one group makes a decision it is final Balazs: who will vote? Mark: everybody has a vote David: voting is wrong split up and work on nos and unclears and clear them Mark: OGF has already solved this Luigi: not possible to split nobody else from gLite Morris: fine with trying this David: remove this voting if somebody does not show up it is their problem Balazs: problem with finding two persons in gLite David: just continue with one group going through the unclears Mark: will channel Andrew we would find people for parallel groups we could do parallel Luigi: maybe gLite can find a second or third person Morris: setting up teams in the google doc Team 1, Team 2, Middleware, Reserve Morris, Shahbaz, UNICORE, Shiraz Balazs, Aleksandr, ARC, Oxana Luigi, ???, gLite, Paolo Etienne, ???, EDGeS, ??? Emannouil, Steve Brewer? , Globus, ??? Steve Crouch, Tim, OMI-UK, David ???, ???, GENESIS, Andrew, John ???, ???, NAREGI, ??? -- _ _ _ _ _ _ Johannes Watzl |\/| |\ | |\/| Institut für Informatik / Dept. of CS | | | \| | | Ludwig-Maximilians-Universität München ======= TEAM ======= Oettingenstr. 67, 80538 Munich, Germany Room E 005, Phone +49-89-2180-9162 Munich Network Management Team Email: watzl@nm.ifi.lmu.de Münchner Netz-Management Team http://www.nm.ifi.lmu.de/~watzl