11:00-12:30 PGI Session 2 ------------------------- Chair: Andrew Grimshaw Minutes: Steve Crouch Attendance: ~16 Name shortcuts: AG - Andrew Grimshaw EU - Etiennce Urbah DK - Daniel Katz OS - Oxana Smirnov JPN - JP Navarro KS - Katsushige Saga SC - Steve Crouch New actions: [BS] Write-up proposal for including benchmarking requirements in JSDL [EU/OS] Write-up proposal for using benchmarks as measurement units in terms of the resource requirements [SC/AG] Another file staging profile++ Minutes: Summary of yesterday's session on agreement of moving forward with existing specs, profiling/updating individual ones and moving towards an overall profile which brings these together. See AG's session presentation for more details. BES++ [See AG's session presentation for more details. List of proposals for last session and open questions.] JSDL Which XML rendering to use for GLUE2 embedded in JSDL, hold before/after execution. Any others? AM: rumour of final version of XML rendering from GLUE group is imminent. [Is it flat or hierarchical?] Can use sub-elements from GLUE2 and include smaller parts. AG: representative from GLUE? AM: JP Navarro good candidate. AG: JSDL's session poorly attended. ActivityManagement Port Type i.e. suspend(), resume(), get_status(), terminate() AG: Idea being that you could have hold states prior to data staging. With this port type, should have other sensible management activies e.g. terminate(). EU: a 'purge' exists in PGI requirements. AG: may get rid of info about the job. 'Activity Integration Profile' AG: profiling for this management port type. e.g. RNS 1.1, OGSA-ByteIO 1.0 (yesterday's session). JPN: ad-hoc GLUE2 session - need understanding of how JSDL wants to use GLUE, to address use cases. 1) allow GLUE2 subelement rendering in other schemas e.g. JSDL; 2) understanding PGI use cases. GLUE2 describes configuration, not matching rules. JSDL may want a, b, c but ranges cannot be described in GLUE2, it's static description. AG: 1) have BES resources use GLUE2 to describe itself; 2) specify job requirements. In JSDL, can specify ranges? AM: how is this matched with the two elements? AG: shortcoming of JSDL - we need to do OR's, but more really guarded statements. We originally want players in JSDL/BES/GLUE to present in these sessions. Do we want to go depth-first? i.e. JSDL first. BES appears straightforward, EU mentioned one way yesterday. Decision on substate model? Then can profile out. EU: can KS provide links to their state model? KS: will provide this this afternoon. OS: user likes to describe job assuming resources offer different configurations, but discovery and matching already done, but when it gets to resource, it's a single item. Do we need logic instructions? AG: JSDL with user submits and what ends up in BES can be different. Important to have consensus on how this is described in JSDL, even if a broker exists in the middle dealing with this translation. User wants to run app, don't care where. May be diff staging thing to do. With ORs? Better with guarded statements with predicates. Can decide based on these. This is better. Guarded statements presented in CSP, incorporated into lots of languages in 80's. OS: implementors - dont know how to implement discovery with logic, through workflows, or deterministic model. Either end of execution service, or in client. AG: request things that have been modified in JSDL from groups e.g. UNICORE. OS: limit JSDL to what arrives at BES. User tasks can describe whatever they like. AG: not a big fan. Would like to take JSDL docs and throw them to EUropean BES'. Not necessary to define these things, but will hurt interop, won't be able to use others' JSDLs and vice versa. OS: broker? AG: EMS can take in documents and transform them along the way. EU: Oxana - do you see BES as a site service, or a machine service? ... AG: would like as symmetric as possible. A site or machine service, but can be entire set of sites or machines. I like recursive layer not being part of design. AG: [on guarded statements] Select (condition expression): action (condition expression): action ... Sepearte our resource req section, but repeating groups of that. For each one have job description, staging elements. On diff machines, want to execute diff apps and move diff data. i.e. diff actions dependent on conditions. Too much of a ++ or BES? We want to have small steps really, instead have smaller pieces. Leave until later? AG: postpone for now, not much enthusiasm for tackling this. EU: inside JSDL, have POSIXApplication. Would like to deprecate it. In same element, it describes subelements on environment, and others belonging to resources. e.g. input/output/error/working directory - really belong in env of execution. Already sep blocks for env and resources in JSDL. Mike: need to be careful careful. BS: +1 on deprecating limits. Mike: places where we specify OS, GLUE for this. We want to still allow JSDL to specify this, and profile to specify this as GLUE2 attributes. EU: but conflict of attributes ... Mike: it complicates the logic. AG: deprecate - not something to count on in future. Take it out of follow-on spec. If moving forward, can take these out, just discourage use in future. Go thru JSDL issues and put them on table. Maybe not make decision yet, but just to list them. BES things fairly well understood. Just JSDL for now. For BES++, what should a BES return on a param sweep [added to slides]. EU: param sweep is dynamic creation of jobs. Not possible to return fixed list of EPRs since created ... ? AG: cardinality of set known at request time. EU: for bulk requests yes, but param sweep number is much higher. AG: need to get to what a BES should return. EU: jobs created one after the other in dynamic way. Supposed that there are too many to list them. AG: our imp generates EPRs from param sweep at the time. Mike: timeout situation? AG: would like to push this back. EU: propose to replace param sweep with vector or bulk. AG: disagree strongly. EU: param sweep is different than vector/bulk. Vector/bulk are finite lists = finite list of EPRs. Param sweep, not finite, but algorithm to dynamically create these. AG: no conditional statements in algorithm, deterministic. Size can be worked out. Open to discussing param sweep on call, but esoteric for now, perhaps not useful. Another JSDL thing (Mark Morgan originally) found resouce factory attributes not enough, sometimes want to match on a property, exec host have requirement that job has attribute. e.g. Kraken, Crays require statically linked binaries. Can'd send dynamically linked program. App needs to say this about itself. So would like to have matching token which has these descriptions for matching on. EU: solved by GLUE? AG: in JSDL ability to describe wanting something, but not predefined e.g. supply as a token. EU: GLUE env entity. AG: can have strings? Corresponding element in the resource description? JPN: app env subelement of other GLUE2 elements, including resource descriptions. AG: could we do - host has BLAST 3.4 and app needs BLAST 3.4 for this. Similarly, relationship for specifying apps as statically linked. JPN: statically linked - haven't heard of this. Perhaps turn this into GLUE2 requirements - not currently adtervised but could be. EU: in GLUE - app name, app version, state of app, can add static desc. in 'Other info'. AG: just a string? e.g. accepts VISA? [Yes] we could include this as we go. BS: difficult - our main thing is having extensible resource model, to have something that allows resources defined by us (not in GLUE/JSDL days) to stay extensible for things we dont know yet. Extension - key/value pairs. AG: can add key/value pairs to GLUE2? JPN? JPN: extension elements all over GLUE - the way to do it. Don't know if formatted as key/valur pairs, perhaps strings. AG: hinge upon things expressed in JSDL. With GLUE, you must have what BES must support, and features for matching. We'll be guinea pigs. EU: good to use benchmarks as measurement units. BS: scalable requirements ... e.g. 2000 CPU hours as benchmark value. AG: normalisation of CPU hours based on benchmark? BS: impossible or very very hard. Benchmark very approximate. AG: scaling factor depends on app. OS: benchmarks may be related to GLUE2, but not JSDL - a valid unit? EU: don't like normalisation or scaling factor. Don't use scalable time. Want to specify 2000 SPECints; it means something. EG: IEEE computer (Freund) - how do we do scaling based on affinities. Very app dependent. you suggest we just use specint family? OS: any. AG: here's a table - this resource has this factor. EU: in GLUE2, can specify. AG: can u point to a benchmark and use it? DK: can go into long list, but best to think of a couple of parameters. AG: really want a performance estimator. Problem is either use simplistic methods, or complicated models. Fine with what ever is desired. DK: needs to be something app env will provide. More complicated, less likely. AM: allowing users to submit benchmarks, nothing could match it. AG: [removing normalisation and benchmarks on slides] - want to specify benchs in terms of what we want. It makes sense. EU: yes. AG: need to come up with extensible list of benchmarks that each have float/int associated with them. EU: already in GLUE. AG: BES endpoint manager advertises that they are e.g. 27 in this benchmark. Request says at least 30. Do we say time I want is based on this? DK: 1) using benchmarks nice, need to say what these are - painful, poss worth it; 2) what do you do with it? AG: break this up into separate standalone profile in JSDL e.g. JSDL Benchmark Profile? OS: depends on application. DK: no, depend on something generic. AG: on benchmark. If extensible, have BLAST benchmark. EU: GLUE bogomips, specint, ... it's extensible. AG: these sorts of benchmarks typically function of machine. EU: open enumeration equivalent of string. AG: this machine, runnig this problem size, ... could be all over the map for tightly coupled systems. Suggest ask those that want it to provide proposal - what would it look like for JSDL and for BES Factory Attributes for how ti would be characterised. At least 2 groups interested - no reason not to profile it, but not as a MUST for resource providers. DK: do res provs have obligation to provide some benchmarks? AG: not mandatory, but profile in terms of how it _can_ be specified. DK: interesting to know how it would be used. AG: yes, people who care about this write this up. e.g. profile, we have use cases, in JSDL do this as a profile and not mandatory in JSDL. EU: go to PGI reqs Wiki and JD20. BS: ideally have a schema for this, but prob not necessary. Have diff models for providing resources and requesting resources. Often not the same. ... AG: action item to write this up? Action: [BS] Write-up proposal for extensible resource model for systems we don't know yet - key/value pairs AG: leave guarded statements for now, but have agreed on many things [see session slides]. Want names to some of these things. Action: [EU/OS] Write-up proposal for using benchmarks as measurement units in terms of the resource requirements EU: should be able to describe diff data sources in sequential-try mode e.g. first fails, second fails, use third (it works). Output file semantics not clear, don't discuss. AG: JSDL is declarative. EU: order doesn't matter. AG: shortcoming of JSDL from users - can't do wildcard staging, including differentiating betwen files and folders. Right now we have File Staging extensions. Do a profile for additional file staging protocols. Action: [SC/AG] Another file staging profile++ AG: related to wild card staging, may want to do -r recursive, inclusion/exclusion patterns. How to prioritise - last 9 mins, get consensus on this. High, medium, low groups. [See AG's session slides for prioritisation list]