May 2010 - pgi-wg - lists.ogf.org

OGF PGI Requirements - Flexibility and clarity versus Rigidity and confusion
by Etienne URBAH 19 Jul '10

19 Jul '10

Balazs, Morris, Luigi, Johannes and all, Concerning the capture and discussion of the requirements : I think that the main problem is NOT the large number of requirements, but the fact that many original requirements were formulated in a very confusing manner. In particular, many original requirements simultaneously present : A) mainly, the WAY the Execution Service would perform a desired functionality, B) sometimes, the INTERFACE which a Client would use to access this functionality, C) implicitly, the NEED for this functionality. This causes us to lose much time in useless discussions, because this requires to use following process : - First, focus on C) to precisely define the functionality, and assess its usefulness, - Second, focus on B) to define the Client interface, - Third, verify that there is no garbage from A) left anymore. Working on the original explicit, implicit or hidden requirements found inside the documents, this is exactly the process which I personally followed, splitting complex confusing requirements into simpler and clearer ones. This permitted that requirements which I personally wrote down could be often quickly understood, discussed, and even sometimes quickly labeled YES (positively agreed) or NO (No agreement yet). For the many requirements which are still unclear, I therefore urge you to accept that I split them into simpler smaller requirements : - Apparently, it will increase the total number of requirements, - But in fact, after an easy work, that will increase the percentage of requirements labeled YES and diminish the percentage of requirements labeled UNCLEAR. As example, please study following requirement, labeled 'Still UNCLEAR after 10 mn discussion' : - JD20 (115) Job Description document should offer a new re-usable structure to define 'scalabletime' requests optionally depending on benchmark results. I will split it into the 4 following requirements : - Description of the needed functionality : JD20.1 For 'Wall clock time' and 'CPU time' limits, the Execution Service SHOULD accept Client requirements independently of the speed of the Computing Resource on which the Activity will run - Description of the conceptual Client interface : JD20.2 For 'Wall clock time' and 'CPU time' limits, the Job Description document SHOULD offer a new re-usable structure permitting to describe requirements independently of the speed of the Computing Resource on which the Activity will run - First option for a precise description of the Client interface : JD20.3 For 'Wall clock time' and 'CPU time' limits, the re-usable structure permitting to describe requirements independently of the speed of the computing resource on which the Activity will run SHOULD be the ordered pair (Value, Unit) where : - 'Value' is the max number of operations allowed for the Activity, - 'Unit' is as unit suitable for a total number of operations, from a closed list to be refined (such as 'Specint2006 * hours' or 'Billions of floating point operations') This method is already used by BOINC and has proved it robustness. - Second option for a precise description of the Client interface : JD20.4 For 'Wall clock time' and 'CPU time' limits, the re-usable structure permitting to describe requirements independently of the speed of the computing resource on which the Activity will run SHOULD be a 'scalabletime' specified by a (Value, BenchmarkType, BenchmarkValue) triplet as described in chapter 7.2 of the Specification Draft I am absolutely certain that these 4 simple requirements will be quickly understood. I am quite sure that we will immediately obtain a YES for JD20.1 and JD20.2, and perhaps quickly a YES to either JD20.3 or JD20.4 (and a NO to the other one). In the same way of thinking, I will re-propose following dropped requirements. With sometimes an improved title, they describe the NEED for a precise Client interface, WITHOUT specifying the precise component which will implement them : - IS10 For requests querying Status of finished Activity, the Information functionality MUST accept, as input parameter, 1 Activity ID - IS12 For requests querying Activity Status, the Information functionality MUST accept, as input parameter, complex queries on Activities expressed according to the GLUE2 model without explicit specification of the Activity IDs. The Information functionality MAY support multiple query languages in addition to the mandatory one. - LB4 For all requests, the Query functionality for Activity Logs MUST accept, as input parameter, 1 Activity ID - LB6 For all requests, the Query functionality for Activity Logs MUST accept, as input parameter, complex queries on Activities expressed according to the GLUE model without explicit specification of the Activity IDs. The Logging and Bookkeeping functionality MAY support multiple query languages in addition to the mandatory one. Besides, I just detected that we forgot to study following requirement, which was mentioned in my mail dated 26 April 2010. Is it possible to associate spreadsheet ID 172 to it ? - IS14 In order to prevent overloading the Execution Service (which has performance requirements for Activity Management), the Information functionality SHOULD be separated from the Execution Service Thank you in advance for accepting that, when starting with original confusion : - rigidity prevents improvement and favors endless confusion, - flexibility favors improvement and then clarity. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------

3 5

Re: [Pgi-wg] PGI Telcon 2010-05-20: Requirements List Spreadsheet - OGF29 + PGI F2F in Chicago - Next telcon
by Etienne URBAH 09 Jun '10

09 Jun '10

Morris, Concerning the OGF PGI telephone conference held on 20 Mai 2010 : - Thank you for leading the discussion. We made much progress. - Could you publish in GridForge the updated MS-Excel source file of the Condensed Requirements List ? - OGF29 + PGI F2F in Chicago : Can you confirm that we can book our travel back home for Friday 25 June 2010 ? - For the next PGI telephone conference : - I heard that there will be NONE next week. - On Thursday 03 June 2010, it would be difficult for me to attend after 16h CET, but I will attend until 17h CET if no other schedule is possible. - Could you propose a date and time ? Thank you in advance. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Thu, 20/05/2010 15:10, Morris Riedel wrote: > Dear PGI team, > > just a short reminder for our meeting today at 3:30 pm CET: > > > (1) > TelCon System using HiDef (Attention: New Number! Not the old PGI one) > > Free of charge when calling from Skype: > +9900827045746500 > > International Numbers: > United States +1 (201) 793-9022 > Canada +1 (201) 793-9022 > Austria +43 (0) 82040115470 > Belgium +32 (0) 7 0357134 > France +33 (0) 826109071 > Germany +49 01805009527 > Ireland +353 (0) 818270968 > Italy +39 848390177 > Spain +34 (9) 02885791 > Switzerland +41 (0) 8 48560397 > United Kingdom +44 (0) 8454018081 > > Room number: 5746500 > > No costs from Skype and nothing to install on the computer. > > > (2) > Screen-Sharing (Voice Backup) > > http://webconf.vc.dfn.de/ogf-pgi-wg/ > > No login is required. Just enter as 'guest' with your real world name. > > > > (3) > Proposed agenda: > > (a) > Sessions at OGF in Chicago > > (b) > Requirement List > > (c) > AOB > > > Please feel free to add other items on the agenda as you like. > > > A high participation from the PGI and GIN members would be appreciated. > > Your co-chair, > Morris > > ------------------------------------------------------------ > Morris Riedel > Distributed Systems and Grid Computing Division > > Jülich Supercomputing Centre (JSC) > Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel

1 2

PGI at OGF?
by Mark Morgan 31 May '10

31 May '10

So, the OGF schedule just came out from Joel and I can't find a PGI session on the schedule? I thought we were going to have a day full of them? Is something wrong or am I missing something? -Mark

2 1

Re: [Pgi-wg] OGF PGI Use Cases and Scope
by Etienne URBAH 21 May '10

21 May '10

Balazs, Morris, Luigi, Johannes and all PGI members, Concerning OGF PGI Uses Cases and Scope : UML diagrams ------------ I have updated inside http://forge.gridforum.org/sf/docman/do/listDocuments/projects.pgi-wg/docma… the source file and the pictures of UML Class and Collaboration Diagrams (designed with the ArgoUML tool) : - showing INTERFACES of Distributed Data Processing which are necessary or useful to standardize, - permitting to assess the impact of SCOPE and architecture on the list of INTERFACES which are absolutely required to be standardized to permit minimum interoperability. For each interface, I have added the known relevant standard(s) inside square brackets. Inside the Collaboration Diagrams, arrows in RED depict relationships NOT using interfaces. These relationships hinder interoperability and should be avoided when possible. Terminology / Vocabulary ------------------------ - INSTRUMENT : Anything generating large sets of scientific data (HEP experiment, Medical imaging, Observation instrument, ...) as taken into account by the DORII project - PAYLOAD : Any Grid or Cloud Entity directly useful for a scientist (Data, Activity, Instrument, ...) - SUPPORT : Any other Grid or Cloud Entity (Security, Info, Application, License, VM image, Log, Accounting, ...) Security, Info, Log and Accounting are absolutely needed to operate a Production Grid. Files ----- http://forge.gridforum.org/sf/go/doc15977?nav=1 ArgoUML source file http://forge.gridforum.org/sf/go/doc15978?nav=1 Class Diagram http://forge.gridforum.org/sf/go/doc15979?nav=1 Abstract Collab. Diagram http://forge.gridforum.org/sf/go/doc15980?nav=1 Detailed Collab. Diagram http://forge.gridforum.org/sf/go/doc15981?nav=1 Scope with BES + JSDL http://forge.gridforum.org/sf/go/doc15982?nav=1 Interoperability with a Monolithic Execution Service Standardization priorities -------------------------- The collaboration diagrams show in particular that : - The interfaces for Activity Management are only connected to the Instrument Manager, the Activity Managers and the Computing Resources. So, it is easy to build a gateway to bridge Activity Management between 2 completely different Grid or Cloud infrastructures (The EDGeS 3G bridge is in full operation between gLite- and BOINC-powered infrastructures). Therefore, Activity Management alone does NOT require urgent standardization. - The interfaces for Security, Info, Log and Accounting are directly connected to most functionalities. Therefore, interoperability between Production Grids absolutely require urgent standardization of these interfaces. Criticism, remarks and suggestions are welcome. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Fri, 14/05/2010 18:03, Etienne URBAH wrote: > Balazs, Morris, Luigi, Johannes and all PGI members, > > > Lot of thanks to Oxana, Aleksandr and Andrew for their contribution to > OGF PGI : > > We finally come to what we all should have begun with, that is the USE > CASES and the SCOPE. > > > +-------------+ > | USE CASES | > +-------------+ > I suggest that : > > - We all provide clear descriptions of Use Cases inside > http://forge.gridforum.org/sf/docman/do/listDocuments/projects.pgi-wg/docma… > > In particular, Use Cases for the five LHC categories described by Oxana > are welcome. > Could the already published GROMACS Use Case be included inside one LHC > category ? > > - Each Use Case (in particular the GROMACS Use Case) must clearly > indicate if the required file stagings should be performed : > - automatically by the Execution Service (from locations specified > inside the Job Description), or > - manually by the Submitter of the Activity, which then requires to be > in a 'Hold' state, and its 'session directory' (or alike) published. > > - We describe these Use Cases with a graphical UML tool, such as ArgoUML > (which runs on MS-Windows, MAC OS X and Linux), and publish the > corresponding source files inside GridForge (open collaborative process). > > - Simple example of Use Case : > The Submitter of an Activity MAY be a scientist using a Scientific Grid > Portal permitting to submit predefined Applications. > Therefore, this Activity Submitter MAY have very little knowledge of > Grids, and MAY have very little knowledge of the Application executed as > Payload on the Computing Resource. > > > +---------+ > | SCOPE | > +---------+ > Execution of Activities or Jobs is only a little part of the > 'Distributed Data Processing' functionalities that a Production Grid is > required to provide. > > In particular, we MUST clearly understand the difference of 'Quality of > Service' required by : > - Transient entities (such as Activities or Jobs), which MAY fail at any > time for any reason, > - Persistent entities (such as Grid resource descriptors, Security > descriptors, Data sets, Accounting Records, Log records, ...), which > SHOULD be securely kept. > > I will publish very soon the source file and the pictures of UML > Collaboration Diagrams (designed with the above mentioned ArgoUML tool) > showing : > > - INTERFACES of Distributed Data Processing which are necessary or > useful to standardize > > - The impact of ARCHITECTURE on the list of INTERFACES which absolutely > require to be standardized to permit minimum interoperability. > > > +------------------------+ > | FOUNDATION STANDARDS | > +------------------------+ > In order to ease mutual understanding and general agreement, we need > Foundation Standards as sound basis. > > Requirement NF6 (162) : JSPG (Security Policies) > ------------------------------------------------- > Without agreement on AUTHN and AUTHZ, there is simply NO interoperability. > > Requirement IS1 (1) : GLUE model > --------------------------------- > - At the Amsterdam meeting, someone said about GLUE : 'Information model > does not concretely say anything'. > - I have the totally opposite opinion : I strongly suggest to use the > GLUE model (currently GLUE 2.0) as one of these Foundation Standards, > and to describe as much concepts as possible using GLUE entities. > > > +----------------------------+ > | TERMINOLOGY / VOCABULARY | > +----------------------------+ > For clarity, I suggest following definitions : > > Client > ------ > Holder of credentials belonging to a member of a GLUE UserDomain. > For example, a Client MAY submit an Activity to an Execution Service, or > query the Status of an Activity. > > Activity = Job (fully synonyms) > -------------- > Remote processing which a Client describes in a 'Job Description', which > the Client then submits to an Execution Service. > > Payload > ------- > Anything (Application, Script, Pilot Job, ...) executed by a Computing > Resource on request of the Activity. The payload MAY completely ignore > that is is executed inside a grid Activity. > > Simple Activity > --------------- > Simple Job Description containing only ONE local job executed by only > ONE batch system, WITHOUT 'Hold' states NOR manual staging. > - This is a suggestion of restrictive evolution of requirement JM5 (55). > - An Activity requiring 'Hold' states and/or manual staging is then > called 'Interactive Activity'. > > > Criticism, remarks and suggestions are welcome. > > Best regards. > > ----------------------------------------------------- > Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS > Bat 200 91898 ORSAY France > Tel: +33 1 64 46 84 87 Skype: etienne.urbah > Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr > ----------------------------------------------------- > > > On 13/05/2010 21:00, Oxana Smirnova wrote: >> Hi Andrew, all, >> >> allow me to start from very starters, to explain "typical" workload >> Aleksandr referred to. >> >> Both ARC and gLite "grew" from the requirements of the High Energy >> Physics community, more specifically - those of LHC experiments. I'll >> come back later to what it means in practice. >> >> The basic difference between gLite and ARC starting requirements is that >> gLite is designed for resources owned or controlled to a large extent by >> their users, while ARC is designed to support resources shared between >> different users and controlled by fairly independent resource providers. >> >> The immediate difference is policies: while gLite community is largely >> expected to comply with policies devised by the LHC Grid (WLCG) Joint >> Security and Policy Group, ARC community has no unique set of policies. >> ARC sites that contribute to LHC computing in general tend to respect >> the WLCG policies, but not too close, giving priority to local policies >> of resource owners. Needless to say, this introduces extra complexity >> into requirements, and reduces the rate of simple use cases. >> >> Now, LHC experiments are huge communities, between 500 and 3000 members >> in each. All well-educated computer-savvy people who never hesitate >> coming with their own brilliant solutions. Even within one experiment >> divergence is huge, and there are sites that support 4 experiments. >> This adds to the complexity: not only we have diverging or contradictory >> requirements from resource owners, we also have all sorts of diverging >> requirements from users. >> The least common denominator is 1: meaning "ping", because even >> "hello world" is ambiguous >> - where do you send the output, to a file or to a standard output? >> If it is standard output, do you write it to a log? >> Who said "hello world", the individual user or the whole experiment? >> Do we have a right to log individual activities at all? And so on. >> >> There are several basic kinds of typical jobs by LHC experiments, in >> general they can be separated in 5 categories: >> >> 1. Monte Carlo generation (no input, small resource consumption, small >> output) >> 2. Detector simulation (one input file, moderate resource consumption, >> moderate output) >> 3. Signal reconstruction (multiple input files, moderate resource >> consumption, large output) >> 4. Data merging (very many input files - like, 400, large resource >> consumption, large output) >> 5. Data analysis (huge amount of input files not necessarily known in >> advance, small resource consumption, small output) >> >> The job can have a number of states, e.g.: >> >> 1. Job is defined (may require authorisation) >> 2. Job matched to a site (requires authorisation, detailed site >> information, maybe data availability information) >> 3. Data are made available to a job (authorisation, probably delegation >> of data staging rights to a staging service) >> 4. Job processes data (CPU, I/O, network access to external databases >> requiring authorisation) >> 5. Job places data at [multiple] pre-defined destinations >> (authorisation, contacting external databases, probably delegation to a >> staging service) >> 6. Job is finished >> 7. Job has failed >> 8. Job is terminated (requires authorisation) >> 9. Job is resubmitted (authorisation, information) >> >> Each state may have a number of sub-states, depending on the >> experiment-specific framework. >> >> Authorisation may be per Virtual Organisation (file reading), per Role >> and/or Group within a VO (job types 1-4), per person (job type 5), or >> even per service (some frameworks accept services as VO members). >> >> Delegation of rights in general is very much needed, because of the >> large number of auxiliary services, distributed environment and general >> complexity of the workflow. No-one really knows how to achieve the goals >> without delegation. >> >> Each experiment has their own framework. Most such frameworks circumvent >> Grid services because they are too generic. This means that jobs are >> highly non-trivial as they attempt to re-implement Grid services such as >> information gathering and publishing, data movement and registration, >> resubmission etc.; they also tend to tweak authorisation by executing >> payload of users not authorised by Grid means. This complicates even >> further the job state picture. >> >> If PGI's outcome will make any of the above mentioned jobs impossible, >> most key ARC and gLite customers will not use PGI specs, and they will >> have only academic value. This was not the PGI goal, as "P" stands for >> "Production". >> >> Cheers, >> Oxana >> >> >> 13.05.2010 15:20, Andrew Grimshaw пишет: >>> Aleksandr, >>> Referring to your sentence/paragraph >>> >>> "Such "simple" job is very far from being "typical". At least in >>> NorduGrid world AFAIK." >>> >>> Could you elaborate. I see in my work basically two "types" of jobs >>> that dominate - sets of HTC "parameter space" jobs, and true parallel >>> MPI jobs. >>> In both cases the "job" is basically a command line - either an >>> mpiexec/run, and application with parameters, or a script with >>> parameters. >>> The job has known inputs and outputs, or a directory tree location >>> where it needs to run. >>> The jobs runs to completion, or it fails, in either case there are >>> output files and result codes. >>> Sometimes the job is a workflow, but when you pick that apart it turns >>> into jobs that have inputs and outputs along with a workflow engine >>> orchestrating it all. >>> >>> What is a typical job that you see? >>> When I say "typical" I mean covers 80% of the jobs. >>> >>> A >>> >>> -----Original Message----- >>> From: Aleksandr Konstantinov [mailto:aleksandr.konstantinov@fys.uio.no] >>> Sent: Sunday, May 02, 2010 3:36 PM >>> To: pgi-wg(a)ogf.org >>> Cc: Andrew Grimshaw; 'Oxana Smirnova'; 'Etienne URBAH'; >>> 'David SNELLING'; lodygens(a)lal.in2p3.fr >>> Subject: Re: [Pgi-wg] OGF PGI Requirements - Flexibility and clarity >>> versus Rigidity and confusion >>> >>> Hello, >>> >>> I agree that problem is to difficult to solve. >>> One should take into account that initially task was different. >>> Originally AFAIR it was an attempt of few grid project to make a >>> common interface suitable for them. Later those were forced to OGF >>> and problem upgraded to almost unsolvable. >>> >>> Andrew Grimshaw at Saturday 01 May 2010 15:36 wrote: >>>> Oxana, >>>> Well said. >>>> >>>> I would add that I fear we may be trying too much to solve all >>>> problems the first time around - to "boil the ocean". >>>> To completely solve the whole problem is a daunting task indeed >>>> as there are so many issues. >>>> >>>> I personally believe we will make more progress if we solve the >>>> minimum problem first, e.g., securely run a simple job from >>>> infrastructure/sw-stack A on infrastructure/sw-stack B. >>> >>> This problem is already solved. And it was done in few ways. >>> 1. Client stacks supporting multiple service stacks >>> 2. BES + GSI >>> 3. Other combinations currently in use >>> And none is fully suitable for real production. >>> So unless task of PGI is considered to be purely theoretical this >>> approach would become equal to one more delay. >>> >>>> >>>> "Infrastructure/sw-stack A" means a set of resources (e.g., true >>>> parallel-Jugene, clusters, sets of desktops) running a middleware >>>> stack (e.g., Unicore 6 or Arc) configured a particular way. >>>> In the European context this might mean an NGI such as D-Grid with >>>> Unicore 6 running a job on NorduGrid running Arc. >>>> (Please forgive me if I have the particulars of the NGIs wrong.) >>>> >>>> "Simple job" means a job that is typical, not special. >>>> This is not to say that its resource requirements are simple, it may >>>> have very particular requirements (cores per socket, interconnect, >>>> memory), rather I mean that the job processing required is simple: >>>> run w/o staging, simple staging, >>> >>> Such "simple" job is very far from being "typical". At least in >>> NorduGrid world AFAIK. >>> >>>> perhaps client interaction with the session directory pre, post, and >>>> during execution. >>>> Try to avoid complex job state models that will be hard to agree on, >>>> and difficult to implement in some environments. >>>> >>>> "Securely" means sufficient authentication information required at B >>>> is provided to B in a form it will accept from a policy perspective. >>>> Further, that we try as much as possible to avoid a delegation >>>> definition that extends inwards beyond the outer boundary of a >>>> particular infrastructure/sw-stack. >>> >>> I'm lost. Is is delegation or definition which extends? >>> >>>> (The last sentence is a bit awkward, I personally think that we will >>>> need to have two models of authentication and delegation >>>> - a legacy transport layer mechanism, >>>> and a message layer mechanism based on SAML, and that inside of a >>>> software stack we cannot expect sw-stacks to change their internal >>>> delegation mechanism.) >>>> >>>> I believe authentication/delegation is the most critical item: >>>> if we cannot get the authentication/delegation issues solved, >>>> the rest is moot with respect to a PRODUCTION environment. >>>> We may be able to do demo's and stunts while punting on >>>> authentication/delegation, but we will not integrate production >>>> systems.) >>> >>> Wasn't delegation voted no during last review? >>> >>> >>> A.K.

1 0

PGI call notes, 2010-05-20
by Johannes Watzl 20 May '10

20 May '10

participants: Morris, Mark, Steve Crouch, Etienne, Johannes, Balazs, Tim, Emannouil Agenda: (a) Sessions at OGF in Chicago discussing restrictions of participants Tuesday as PGI-day at OGF29 PGI f2f meeting 23-24 June, same venue ARC will find a person, Balasz not sure to go nobody from Southhampton will go to OGF this time action for Morris: contact gLite people in terms of OGF29 planning (b) Requirement List 157: NSF requirement Etienne: could be a spof or bottleneck Mark: not a spof Balazs: interesting but not sure if in scope of pgi so no Morris: not a quick agreement, put a "no" change to "no" ### 158: Morris: NSF requirement, who has a problem with this? Steve: more general, not real requirment Morris: would say it is a duplicate of 157 marked as duplicate ### 159: GIN requirement Balazs: logging and bookkeeping originally: requirement form Czech Grid people Etienne: very much SOAP oriented, would like another formulation Morris: vote for "yes" changed to "yes" ### 160: Mark: basic computer science principle Morris: someone saying "no"? nobody changed to "yes" ### 163: changed to "no" ### 164: Morris: someone saying "no"? vote for "no" changed to "no" ### 165: changed to "yes" ### 166: Morris: information functionality inside the execution service? Etienne: for me different Morris: somebody still problem in unerstanding? someone saying "no" changed to "yes" ### 170: Morris: why reproposed? Etienne: payload: anything that the job submitter wants to exectue on the computer resource, app, script or pilot job here the important thing is to define an abstract model without giving precise names to the states Morris: unclear for somebody? Etienne: "no" for me changed to "no" ### 13: Etienne: could be at the message layer consider authentication at message level? what is the set of the certification authorities? Morris: the what is not the question here. how we do it will be discussed later. now just resolving "unlcears" diving downin security details will discuss security suggestion: mark the four security requirements as "no" changed 13,14,15,16 as "no" ### Etienne: now payload is defined 123,124 could be resolved easily 123: Etienne: payload widely used in other contexts to describe what is the interesting part with real value changed to "yes" ### 124: changed to "yes" ### Morris: how to proceed? Etienne: begin again with beginning of the list and look at all still unclear and no s and try to resolve them try to clarify the whole situation by clearifying the scope of the execution service ### insert requirement 172: IS14 changed to "yes" ### insert requirement 173: changed to "yes" ### (c) AOB -- _ _ _ _ _ _ Johannes Watzl |\/| |\ | |\/| Institut für Informatik / Dept. of CS | | | \| | | Ludwig-Maximilians-Universität München ======= TEAM ======= Oettingenstr. 67, 80538 Munich, Germany Room E 005, Phone +49-89-2180-9162 Munich Network Management Team Email: watzl(a)nm.ifi.lmu.de Münchner Netz-Management Team http://www.nm.ifi.lmu.de/~watzl

1 0

Reminder & Agenda: PGI Telcon Today 2010-05-20; 3:30 pm CET
by Morris Riedel 20 May '10

20 May '10

Dear PGI team, just a short reminder for our meeting today at 3:30 pm CET: (1) TelCon System using HiDef (Attention: New Number! Not the old PGI one) Free of charge when calling from Skype: +9900827045746500 International Numbers: United States +1 (201) 793-9022 Canada +1 (201) 793-9022 Austria +43 (0) 82040115470 Belgium +32 (0) 7 0357134 France +33 (0) 826109071 Germany +49 01805009527 Ireland +353 (0) 818270968 Italy +39 848390177 Spain +34 (9) 02885791 Switzerland +41 (0) 8 48560397 United Kingdom +44 (0) 8454018081 Room number: 5746500 No costs from Skype and nothing to install on the computer. (2) Screen-Sharing (Voice Backup) http://webconf.vc.dfn.de/ogf-pgi-wg/ No login is required. Just enter as 'guest' with your real world name. (3) Proposed agenda: (a) Sessions at OGF in Chicago (b) Requirement List (c) AOB Please feel free to add other items on the agenda as you like. A high participation from the PGI and GIN members would be appreciated. Your co-chair, Morris ------------------------------------------------------------ Morris Riedel Distributed Systems and Grid Computing Division Jülich Supercomputing Centre (JSC) Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel

1 0

Re: [Pgi-wg] OGF PGI Use Cases and Scpoe
by Etienne URBAH 14 May '10

14 May '10

Balazs, Morris, Luigi, Johannes and all PGI members, Concerning OGF PGI Uses Cases and Scope : UML diagrams ------------ As announced, I just published inside http://forge.gridforum.org/sf/docman/do/listDocuments/projects.pgi-wg/docma… the source file and the pictures of UML Class and Collaboration Diagrams (designed with the ArgoUML tool) : - showing INTERFACES of Distributed Data Processing which are necessary or useful to standardize, - permitting to assess the impact of ARCHITECTURE on the list of INTERFACES which absolutely require to be standardized to permit minimum interoperability. For each interface, I have added the known relevant standard(s) inside square brackets. Please note that Accounting Backend is standardized by 'UR' (I just forgot it). Inside the Collaboration Diagrams, arrows in RED depict relationships NOT using interfaces. These relationships hinder interoperability and should be avoided when possible. Terminology / Vocabulary ------------------------ - PAYLOAD : Any Grid or Cloud Entity directly useful for a scientist (Data, Activity, Instrument, ...) - SUPPORT : Any other Grid or Cloud Entity (Security, Info, Application, License, VM image, Log, Accounting, ...) Security, Info, Log and Accounting are absolutely needed to operate a Production Grid. Files ----- http://forge.gridforum.org/sf/go/doc15977?nav=1 ArgoUML source file http://forge.gridforum.org/sf/go/doc15978?nav=1 Class Diagram http://forge.gridforum.org/sf/go/doc15979?nav=1 Small Collab. Diagram http://forge.gridforum.org/sf/go/doc15980?nav=1 Big Collab. Diagram http://forge.gridforum.org/sf/go/doc15981?nav=1 Scope with BES + JSDL http://forge.gridforum.org/sf/go/doc15982?nav=1 Interoperability with a Monolithic Execution Service Standardization priorities -------------------------- The collaboration diagrams show in particular that : - The interfaces for Activity Management are only connected to the Activity Managers and the Computing Resources. So, it is easy to build a gateway to bridge Activity Management between 2 completely different Grid or Cloud infrastructures (The EDGeS 3G bridge is in full operation between gLite- and BOINC-powered infrastructures). Therefore, Activity Management alone does NOT require urgent standardization. - The interfaces for Security, Info, Log and Accounting are directly connected to most functionalities. Therefore, interoperability between Production Grids absolutely require urgent standardization of these interfaces. Criticism, remarks and suggestions are welcome. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Fri, 14/05/2010 18:03, Etienne URBAH wrote: > Balazs, Morris, Luigi, Johannes and all PGI members, > > > Lot of thanks to Oxana, Aleksandr and Andrew for their contribution to > OGF PGI : > > We finally come to what we all should have begun with, that is the USE > CASES and the SCOPE. > > > +-------------+ > | USE CASES | > +-------------+ > I suggest that : > > - We all provide clear descriptions of Use Cases inside > http://forge.gridforum.org/sf/docman/do/listDocuments/projects.pgi-wg/docma… > > In particular, Use Cases for the five LHC categories described by Oxana > are welcome. > Could the already published GROMACS Use Case be included inside one LHC > category ? > > - Each Use Case (in particular the GROMACS Use Case) must clearly > indicate if the required file stagings should be performed : > - automatically by the Execution Service (from locations specified > inside the Job Description), or > - manually by the Submitter of the Activity, which then requires to be > in a 'Hold' state, and its 'session directory' (or alike) published. > > - We describe these Use Cases with a graphical UML tool, such as ArgoUML > (which runs on MS-Windows, MAC OS X and Linux), and publish the > corresponding source files inside GridForge (open collaborative process). > > - Simple example of Use Case : > The Submitter of an Activity MAY be a scientist using a Scientific Grid > Portal permitting to submit predefined Applications. > Therefore, this Activity Submitter MAY have very little knowledge of > Grids, and MAY have very little knowledge of the Application executed as > Payload on the Computing Resource. > > > +---------+ > | SCOPE | > +---------+ > Execution of Activities or Jobs is only a little part of the > 'Distributed Data Processing' functionalities that a Production Grid is > required to provide. > > In particular, we MUST clearly understand the difference of 'Quality of > Service' required by : > - Transient entities (such as Activities or Jobs), which MAY fail at any > time for any reason, > - Persistent entities (such as Grid resource descriptors, Security > descriptors, Data sets, Accounting Records, Log records, ...), which > SHOULD be securely kept. > > I will publish very soon the source file and the pictures of UML > Collaboration Diagrams (designed with the above mentioned ArgoUML tool) > showing : > > - INTERFACES of Distributed Data Processing which are necessary or > useful to standardize > > - The impact of ARCHITECTURE on the list of INTERFACES which absolutely > require to be standardized to permit minimum interoperability. > > > +------------------------+ > | FOUNDATION STANDARDS | > +------------------------+ > In order to ease mutual understanding and general agreement, we need > Foundation Standards as sound basis. > > Requirement NF6 (162) : JSPG (Security Policies) > ------------------------------------------------- > Without agreement on AUTHN and AUTHZ, there is simply NO interoperability. > > Requirement IS1 (1) : GLUE model > --------------------------------- > - At the Amsterdam meeting, someone said about GLUE : 'Information model > does not concretely say anything'. > - I have the totally opposite opinion : I strongly suggest to use the > GLUE model (currently GLUE 2.0) as one of these Foundation Standards, > and to describe as much concepts as possible using GLUE entities. > > > +----------------------------+ > | TERMINOLOGY / VOCABULARY | > +----------------------------+ > For clarity, I suggest following definitions : > > Client > ------ > Holder of credentials belonging to a member of a GLUE UserDomain. > For example, a Client MAY submit an Activity to an Execution Service, or > query the Status of an Activity. > > Activity = Job (fully synonyms) > -------------- > Remote processing which a Client describes in a 'Job Description', which > the Client then submits to an Execution Service. > > Payload > ------- > Anything (Application, Script, Pilot Job, ...) executed by a Computing > Resource on request of the Activity. The payload MAY completely ignore > that is is executed inside a grid Activity. > > Simple Activity > --------------- > Simple Job Description containing only ONE local job executed by only > ONE batch system, WITHOUT 'Hold' states NOR manual staging. > - This is a suggestion of restrictive evolution of requirement JM5 (55). > - An Activity requiring 'Hold' states and/or manual staging is then > called 'Interactive Activity'. > > > Criticism, remarks and suggestions are welcome. > > Best regards. > > ----------------------------------------------------- > Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS > Bat 200 91898 ORSAY France > Tel: +33 1 64 46 84 87 Skype: etienne.urbah > Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr > ----------------------------------------------------- > > > On 13/05/2010 21:00, Oxana Smirnova wrote: >> Hi Andrew, all, >> >> allow me to start from very starters, to explain "typical" workload >> Aleksandr referred to. >> >> Both ARC and gLite "grew" from the requirements of the High Energy >> Physics community, more specifically - those of LHC experiments. I'll >> come back later to what it means in practice. >> >> The basic difference between gLite and ARC starting requirements is that >> gLite is designed for resources owned or controlled to a large extent by >> their users, while ARC is designed to support resources shared between >> different users and controlled by fairly independent resource providers. >> >> The immediate difference is policies: while gLite community is largely >> expected to comply with policies devised by the LHC Grid (WLCG) Joint >> Security and Policy Group, ARC community has no unique set of policies. >> ARC sites that contribute to LHC computing in general tend to respect >> the WLCG policies, but not too close, giving priority to local policies >> of resource owners. Needless to say, this introduces extra complexity >> into requirements, and reduces the rate of simple use cases. >> >> Now, LHC experiments are huge communities, between 500 and 3000 members >> in each. All well-educated computer-savvy people who never hesitate >> coming with their own brilliant solutions. Even within one experiment >> divergence is huge, and there are sites that support 4 experiments. >> This adds to the complexity: not only we have diverging or contradictory >> requirements from resource owners, we also have all sorts of diverging >> requirements from users. >> The least common denominator is 1: meaning "ping", because even >> "hello world" is ambiguous >> - where do you send the output, to a file or to a standard output? >> If it is standard output, do you write it to a log? >> Who said "hello world", the individual user or the whole experiment? >> Do we have a right to log individual activities at all? And so on. >> >> There are several basic kinds of typical jobs by LHC experiments, in >> general they can be separated in 5 categories: >> >> 1. Monte Carlo generation (no input, small resource consumption, small >> output) >> 2. Detector simulation (one input file, moderate resource consumption, >> moderate output) >> 3. Signal reconstruction (multiple input files, moderate resource >> consumption, large output) >> 4. Data merging (very many input files - like, 400, large resource >> consumption, large output) >> 5. Data analysis (huge amount of input files not necessarily known in >> advance, small resource consumption, small output) >> >> The job can have a number of states, e.g.: >> >> 1. Job is defined (may require authorisation) >> 2. Job matched to a site (requires authorisation, detailed site >> information, maybe data availability information) >> 3. Data are made available to a job (authorisation, probably delegation >> of data staging rights to a staging service) >> 4. Job processes data (CPU, I/O, network access to external databases >> requiring authorisation) >> 5. Job places data at [multiple] pre-defined destinations >> (authorisation, contacting external databases, probably delegation to a >> staging service) >> 6. Job is finished >> 7. Job has failed >> 8. Job is terminated (requires authorisation) >> 9. Job is resubmitted (authorisation, information) >> >> Each state may have a number of sub-states, depending on the >> experiment-specific framework. >> >> Authorisation may be per Virtual Organisation (file reading), per Role >> and/or Group within a VO (job types 1-4), per person (job type 5), or >> even per service (some frameworks accept services as VO members). >> >> Delegation of rights in general is very much needed, because of the >> large number of auxiliary services, distributed environment and general >> complexity of the workflow. No-one really knows how to achieve the goals >> without delegation. >> >> Each experiment has their own framework. Most such frameworks circumvent >> Grid services because they are too generic. This means that jobs are >> highly non-trivial as they attempt to re-implement Grid services such as >> information gathering and publishing, data movement and registration, >> resubmission etc.; they also tend to tweak authorisation by executing >> payload of users not authorised by Grid means. This complicates even >> further the job state picture. >> >> If PGI's outcome will make any of the above mentioned jobs impossible, >> most key ARC and gLite customers will not use PGI specs, and they will >> have only academic value. This was not the PGI goal, as "P" stands for >> "Production". >> >> Cheers, >> Oxana >> >> >> 13.05.2010 15:20, Andrew Grimshaw пишет: >>> Aleksandr, >>> Referring to your sentence/paragraph >>> >>> "Such "simple" job is very far from being "typical". At least in >>> NorduGrid world AFAIK." >>> >>> Could you elaborate. I see in my work basically two "types" of jobs >>> that dominate - sets of HTC "parameter space" jobs, and true parallel >>> MPI jobs. >>> In both cases the "job" is basically a command line - either an >>> mpiexec/run, and application with parameters, or a script with >>> parameters. >>> The job has known inputs and outputs, or a directory tree location >>> where it needs to run. >>> The jobs runs to completion, or it fails, in either case there are >>> output files and result codes. >>> Sometimes the job is a workflow, but when you pick that apart it turns >>> into jobs that have inputs and outputs along with a workflow engine >>> orchestrating it all. >>> >>> What is a typical job that you see? >>> When I say "typical" I mean covers 80% of the jobs. >>> >>> A >>> >>> -----Original Message----- >>> From: Aleksandr Konstantinov [mailto:aleksandr.konstantinov@fys.uio.no] >>> Sent: Sunday, May 02, 2010 3:36 PM >>> To: pgi-wg(a)ogf.org >>> Cc: Andrew Grimshaw; 'Oxana Smirnova'; 'Etienne URBAH'; >>> 'David SNELLING'; lodygens(a)lal.in2p3.fr >>> Subject: Re: [Pgi-wg] OGF PGI Requirements - Flexibility and clarity >>> versus Rigidity and confusion >>> >>> Hello, >>> >>> I agree that problem is to difficult to solve. >>> One should take into account that initially task was different. >>> Originally AFAIR it was an attempt of few grid project to make a >>> common interface suitable for them. Later those were forced to OGF >>> and problem upgraded to almost unsolvable. >>> >>> Andrew Grimshaw at Saturday 01 May 2010 15:36 wrote: >>>> Oxana, >>>> Well said. >>>> >>>> I would add that I fear we may be trying too much to solve all >>>> problems the first time around - to "boil the ocean". >>>> To completely solve the whole problem is a daunting task indeed >>>> as there are so many issues. >>>> >>>> I personally believe we will make more progress if we solve the >>>> minimum problem first, e.g., securely run a simple job from >>>> infrastructure/sw-stack A on infrastructure/sw-stack B. >>> >>> This problem is already solved. And it was done in few ways. >>> 1. Client stacks supporting multiple service stacks >>> 2. BES + GSI >>> 3. Other combinations currently in use >>> And none is fully suitable for real production. >>> So unless task of PGI is considered to be purely theoretical this >>> approach would become equal to one more delay. >>> >>>> >>>> "Infrastructure/sw-stack A" means a set of resources (e.g., true >>>> parallel-Jugene, clusters, sets of desktops) running a middleware >>>> stack (e.g., Unicore 6 or Arc) configured a particular way. >>>> In the European context this might mean an NGI such as D-Grid with >>>> Unicore 6 running a job on NorduGrid running Arc. >>>> (Please forgive me if I have the particulars of the NGIs wrong.) >>>> >>>> "Simple job" means a job that is typical, not special. >>>> This is not to say that its resource requirements are simple, it may >>>> have very particular requirements (cores per socket, interconnect, >>>> memory), rather I mean that the job processing required is simple: >>>> run w/o staging, simple staging, >>> >>> Such "simple" job is very far from being "typical". At least in >>> NorduGrid world AFAIK. >>> >>>> perhaps client interaction with the session directory pre, post, and >>>> during execution. >>>> Try to avoid complex job state models that will be hard to agree on, >>>> and difficult to implement in some environments. >>>> >>>> "Securely" means sufficient authentication information required at B >>>> is provided to B in a form it will accept from a policy perspective. >>>> Further, that we try as much as possible to avoid a delegation >>>> definition that extends inwards beyond the outer boundary of a >>>> particular infrastructure/sw-stack. >>> >>> I'm lost. Is is delegation or definition which extends? >>> >>>> (The last sentence is a bit awkward, I personally think that we will >>>> need to have two models of authentication and delegation >>>> - a legacy transport layer mechanism, >>>> and a message layer mechanism based on SAML, and that inside of a >>>> software stack we cannot expect sw-stacks to change their internal >>>> delegation mechanism.) >>>> >>>> I believe authentication/delegation is the most critical item: >>>> if we cannot get the authentication/delegation issues solved, >>>> the rest is moot with respect to a PRODUCTION environment. >>>> We may be able to do demo's and stunts while punting on >>>> authentication/delegation, but we will not integrate production >>>> systems.) >>> >>> Wasn't delegation voted no during last review? >>> >>> >>> A.K.

1 0

Re: [Pgi-wg] OGF PGI Scope
by Etienne URBAH 14 May '10

14 May '10

Aleksandr, Concerning OGF PGI Uses Cases and Scope : Complexity ---------- The 'requirement for pre-installed complex software packages to add more to complexity' is ALREADY there : This is AR1 (34). Scope ----- In fact, execution of Activities or Jobs is only a little part of the 'Distributed Data Processing' functionalities that a Production Grid is required to provide. In particular, we MUST clearly understand the difference of 'Quality of Service' required by : - Transient entities (such as Activities or Jobs), which MAY fail at any time for any reason, - Persistent entities (such as Grid resource descriptors, Security descriptors, Data sets, Accounting Records, Log records, ...), which SHOULD be securely kept. I just published at http://forge.gridforum.org/sf/go/doc15977?nav=1 the source file of UML Class and Collaboration Diagrams (designed with the ArgoUML tool) : - showing INTERFACES of Distributed Data Processing which are necessary or useful to standardize, - permitting to asses the impact of ARCHITECTURE on the list of INTERFACES which absolutely require to be standardized to permit minimum interoperability. I will publish the corresponding pictures very soon. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Fri, 14/05/2010 20:50, Aleksandr Konstantinov wrote: > Well done, Oxana. Very thorough description. I coudln't say better. > Just add requirement for pre-installed complex software packages > to add more to complexity. > > A.K. > > > > Oxana Smirnova at Thursday 13 May 2010 22:00 wrote: >> Hi Andrew, all, >> >> allow me to start from very starters, to explain "typical" workload >> Aleksandr referred to. >> >> Both ARC and gLite "grew" from the requirements of the High Energy Physics >> community, more specifically - those of LHC experiments. I'll come back >> later to what it means in practice. >> >> The basic difference between gLite and ARC starting requirements is that >> gLite is designed for resources owned or controlled to a large extent by >> their users, while ARC is designed to support resources shared between >> different users and controlled by fairly independent resource providers. >> >> The immediate difference is policies: while gLite community is largely >> expected to comply with policies devised by the LHC Grid (WLCG) Joint >> Security and Policy Group, ARC community has no unique set of policies. ARC >> sites that contribute to LHC computing in general tend to respect the WLCG >> policies, but not too close, giving priority to local policies of resource >> owners. Needless to say, this introduces extra complexity into >> requirements, and reduces the rate of simple use cases. >> >> Now, LHC experiments are huge communities, between 500 and 3000 members in >> each. All well-educated computer-savvy people who never hesitate coming >> with their own brilliant solutions. Even within one experiment divergence >> is huge, and there are sites that support 4 experiments. This adds to the >> complexity: not only we have diverging or contradictory requirements from >> resource owners, we also have all sorts of diverging requirements from >> users. The least common denominator is 1: meaning "ping", because even >> "hello world" is ambiguous - where do you send the output, to a file or to >> a standard output? If it is standard output, do you write it to a log? Who >> said "hello world", the individual user or the whole experiment? Do we have >> a right to log individual activities at all? And so on. >> >> There are several basic kinds of typical jobs by LHC experiments, in >> general they can be separated in 5 categories: >> >> 1. Monte Carlo generation (no input, small resource consumption, small >> output) 2. Detector simulation (one input file, moderate resource >> consumption, moderate output) 3. Signal reconstruction (multiple input >> files, moderate resource consumption, large output) 4. Data merging (very >> many input files - like, 400, large resource consumption, large output) 5. >> Data analysis (huge amount of input files not necessarily known in advance, >> small resource consumption, small output) >> >> The job can have a number of states, e.g.: >> >> 1. Job is defined (may require authorisation) >> 2. Job matched to a site (requires authorisation, detailed site >> information, maybe data availability information) 3. Data are made >> available to a job (authorisation, probably delegation of data staging >> rights to a staging service) 4. Job processes data (CPU, I/O, network >> access to external databases requiring authorisation) 5. Job places data at >> [multiple] pre-defined destinations (authorisation, contacting external >> databases, probably delegation to a staging service) 6. Job is finished >> 7. Job has failed >> 8. Job is terminated (requires authorisation) >> 9. Job is resubmitted (authorisation, information) >> >> Each state may have a number of sub-states, depending on the >> experiment-specific framework. >> >> Authorisation may be per Virtual Organisation (file reading), per Role >> and/or Group within a VO (job types 1-4), per person (job type 5), or even >> per service (some frameworks accept services as VO members). >> >> Delegation of rights in general is very much needed, because of the large >> number of auxiliary services, distributed environment and general >> complexity of the workflow. No-one really knows how to achieve the goals >> without delegation. >> >> Each experiment has their own framework. Most such frameworks circumvent >> Grid services because they are too generic. This means that jobs are highly >> non-trivial as they attempt to re-implement Grid services such as >> information gathering and publishing, data movement and registration, >> resubmission etc.; they also tend to tweak authorisation by executing >> payload of users not authorised by Grid means. This complicates even >> further the job state picture. >> >> If PGI's outcome will make any of the above mentioned jobs impossible, most >> key ARC and gLite customers will not use PGI specs, and they will have only >> academic value. This was not the PGI goal, as "P" stands for "Production". >> >> Cheers, >> Oxana >> >> 13.05.2010 15:20, Andrew Grimshaw пишет: >>> Aleksandr, >>> Referring to your sentence/paragraph >>> >>> "Such "simple" job is very far from being "typical". At least in >>> NorduGrid world AFAIK." >>> >>> Could you elaborate. I see in my work basically two "types" of jobs that >>> dominate - sets of HTC "parameter space" jobs, and true parallel MPI >>> jobs. In both cases the "job" is basically a command line - either an >>> mpiexec/run, and application with parameters, or a script with >>> parameters. The job has known inputs and outputs, or a directory tree >>> location where it needs to run. The jobs runs to completion, or it fails, >>> in either case there are output files and result codes. Sometimes the job >>> is a workflow, but when you pick that apart it turns into jobs that have >>> inputs and outputs along with a workflow engine orchestrating it all. >>> >>> What is a typical job that you see? When I say "typical" I mean covers >>> 80% of the jobs. >>> >>> A >>> >>> -----Original Message----- >>> From: Aleksandr Konstantinov [mailto:aleksandr.konstantinov@fys.uio.no] >>> Sent: Sunday, May 02, 2010 3:36 PM >>> To: pgi-wg(a)ogf.org >>> Cc: Andrew Grimshaw; 'Oxana Smirnova'; 'Etienne URBAH'; 'David SNELLING'; >>> lodygens(a)lal.in2p3.fr >>> Subject: Re: [Pgi-wg] OGF PGI Requirements - Flexibility and clarity >>> versus Rigidity and confusion >>> >>> Hello, >>> >>> I agree that problem is to difficult to solve. One should take into >>> account that initially task was different. Originally AFAIR it was >>> an attempt of few grid project to make a common interface suitable >>> for them. Later those were forced to OGF and problem upgraded >>> to almost unsolvable. >>> >>> Andrew Grimshaw at Saturday 01 May 2010 15:36 wrote: >>>> Oxana, >>>> Well said. >>>> >>>> I would add that I fear we may be trying too much to solve all problems >>> >>> the >>> >>>> first time around - to "boil the ocean". To completely solve the whole >>>> problem is a daunting task indeed as there are so many issues. >>>> >>>> I personally believe we will make more progress if we solve the minimum >>>> problem first, e.g., securely run a simple job from >>> >>> infrastructure/sw-stack >>> >>>> A on infrastructure/sw-stack B. >>> >>> This problem is already solved. And it was done in few ways. >>> 1. Client stacks supporting multiple service stacks >>> 2. BES + GSI >>> 3. Other combinations currently in use >>> And none is fully suitable for real production. So unless task of PGI >>> is considered to be purely theoretical this approach would become >>> equal to one more delay. >>> >>>> "Infrastructure/sw-stack A" means a set of resources (e.g., true >>>> parallel-Jugene, clusters, sets of desktops) running a middleware stack >>>> (e.g., Unicore 6 or Arc) configured a particular way. In the European >>>> context this might mean an NGI such as D-Grid with Unicore 6 running a >>>> job on NorduGrid running Arc. (Please forgive me if I have the >>>> particulars of the NGIs wrong.) >>>> >>>> "Simple job" means a job that is typical, not special. This is not to >>>> say that its resource requirements are simple, it may have very >>>> particular requirements (cores per socket, interconnect, memory), rather >>>> I mean that the job processing required is simple: run w/o staging, >>>> simple staging, >>> >>> Such "simple" job is very far from being "typical". At least in NorduGrid >>> world AFAIK. >>> >>>> perhaps client interaction with the session directory pre, post, and >>> >>> during >>> >>>> execution. >>>> Try to avoid complex job state models that will be hard to agree >>>> on, and difficult to implement in some environments. >>>> >>>> "Securely" means sufficient authentication information required at B is >>>> provided to B in a form it will accept from a policy perspective. >>>> Further, that we try as much as possible to avoid a delegation >>>> definition that extends inwards beyond the outer boundary of a >>>> particular >>> >>> I'm lost. Is is delegation or definition which extends? >>> >>>> infrastructure/sw-stack. (The last sentence is a bit awkward, I >>>> personally think that we will need to have two models of authentication >>>> and >>> >>> delegation >>> >>>> - a legacy transport layer mechanism, and a message layer mechanism >>>> based on SAML, and that inside of a software stack we cannot expect >>>> sw-stacks to change their internal delegation mechanism.) >>>> >>>> I believe authentication/delegation is the most critical item: if we >>> >>> cannot >>> >>>> get the authentication/delegation issues solved, the rest is moot with >>>> respect to a PRODUCTION environment. We may be able to do demo's and >>> >>> stunts >>> >>>> while punting on authentication/delegation, but we will not integrate >>>> production systems.) >>> >>> Wasn't delegation voted no during last review? >>> >>> >>> A.K.

1 0

Re: [Pgi-wg] OGF PGI Requirements - Flexibility and clarity versus Rigidity and confusion
by Andrew Grimshaw 13 May '10

13 May '10

Aleksandr, Referring to your sentence/paragraph "Such "simple" job is very far from being "typical". At least in NorduGrid world AFAIK." Could you elaborate. I see in my work basically two "types" of jobs that dominate - sets of HTC "parameter space" jobs, and true parallel MPI jobs. In both cases the "job" is basically a command line - either an mpiexec/run, and application with parameters, or a script with parameters. The job has known inputs and outputs, or a directory tree location where it needs to run. The jobs runs to completion, or it fails, in either case there are output files and result codes. Sometimes the job is a workflow, but when you pick that apart it turns into jobs that have inputs and outputs along with a workflow engine orchestrating it all. What is a typical job that you see? When I say "typical" I mean covers 80% of the jobs. A -----Original Message----- From: Aleksandr Konstantinov [mailto:aleksandr.konstantinov@fys.uio.no] Sent: Sunday, May 02, 2010 3:36 PM To: pgi-wg(a)ogf.org Cc: Andrew Grimshaw; 'Oxana Smirnova'; 'Etienne URBAH'; 'David SNELLING'; lodygens(a)lal.in2p3.fr Subject: Re: [Pgi-wg] OGF PGI Requirements - Flexibility and clarity versus Rigidity and confusion Hello, I agree that problem is to difficult to solve. One should take into account that initially task was different. Originally AFAIR it was an attempt of few grid project to make a common interface suitable for them. Later those were forced to OGF and problem upgraded to almost unsolvable. Andrew Grimshaw at Saturday 01 May 2010 15:36 wrote: > Oxana, > Well said. > > I would add that I fear we may be trying too much to solve all problems the > first time around - to "boil the ocean". To completely solve the whole > problem is a daunting task indeed as there are so many issues. > > I personally believe we will make more progress if we solve the minimum > problem first, e.g., securely run a simple job from infrastructure/sw-stack > A on infrastructure/sw-stack B. This problem is already solved. And it was done in few ways. 1. Client stacks supporting multiple service stacks 2. BES + GSI 3. Other combinations currently in use And none is fully suitable for real production. So unless task of PGI is considered to be purely theoretical this approach would become equal to one more delay. > > "Infrastructure/sw-stack A" means a set of resources (e.g., true > parallel-Jugene, clusters, sets of desktops) running a middleware stack > (e.g., Unicore 6 or Arc) configured a particular way. In the European > context this might mean an NGI such as D-Grid with Unicore 6 running a job > on NorduGrid running Arc. (Please forgive me if I have the particulars of > the NGIs wrong.) > > "Simple job" means a job that is typical, not special. This is not to say > that its resource requirements are simple, it may have very particular > requirements (cores per socket, interconnect, memory), rather I mean that > the job processing required is simple: run w/o staging, simple staging, Such "simple" job is very far from being "typical". At least in NorduGrid world AFAIK. > perhaps client interaction with the session directory pre, post, and during > execution. > Try to avoid complex job state models that will be hard to agree > on, and difficult to implement in some environments. > > "Securely" means sufficient authentication information required at B is > provided to B in a form it will accept from a policy perspective. Further, > that we try as much as possible to avoid a delegation definition that > extends inwards beyond the outer boundary of a particular I'm lost. Is is delegation or definition which extends? > infrastructure/sw-stack. (The last sentence is a bit awkward, I personally > think that we will need to have two models of authentication and delegation > - a legacy transport layer mechanism, and a message layer mechanism based > on SAML, and that inside of a software stack we cannot expect sw-stacks to > change their internal delegation mechanism.) > > I believe authentication/delegation is the most critical item: if we cannot > get the authentication/delegation issues solved, the rest is moot with > respect to a PRODUCTION environment. We may be able to do demo's and stunts > while punting on authentication/delegation, but we will not integrate > production systems.) Wasn't delegation voted no during last review? A.K.

2 1

OGF29 Chicago 20-22 June 2010 : OGF PGI Sessions
by Etienne URBAH 12 May '10

12 May '10

Balazs, Morris, Luigi and Johannes, For OGF29 in Chicago on 20-22 June 2010, I am planning to attend : - Whole Sunday 20 - Whole Monday 21 - Only morning of Tuesday 22 Can you organize the PGI sessions so that none is scheduled on Tuesday 22 afternoon ? If one or several PGI sessions still have to be scheduled on Tuesday 22 afternoon, please let me know very quickly, so that I prepare my trip to stay in Chicago 1 night more. Thank you in advance. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------

1 0