- jsdl-wg - lists.ogf.org

Limit Text
by Donal K. Fellows 26 Apr '05

26 Apr '05

The following is my suggested text for the MemoryLimit and PipeSizeLimit element definitions. The text in [brackets] is non-normative. I have omitted the text stating what it must be a sub-element of (according to the principles of allowing the schema to be used as an ontology as well as a document definition) but if we really want that, it's trivial to add it. MemoryLimit: Definition This element is a positive integer that describes the maximum amount of physical memory (in bytes) that the job should use when executing. It MUST be supported. If this is not present, then it is unlimited. [This is really the Resident Set Size limit, and systems might impose a granularity of a kilobyte.] PipeSizeLimit: Definition This element is a positive integer that describes the maximum size of pipelines (in bytes) created for and by the processing of the job. It MUST be supported. If this is not present, then the system should use its default value. [No system permits arbitrary sized pipe buffers, but their defaults are usually entirely acceptable. Systems typically impose a granularity of 512 bytes.] Donal.

1 0

Coupling PosixApplication to DataStaging
by Donal K. Fellows 26 Apr '05

26 Apr '05

Hi everyone! We have something of a problem in that we need a way to join the virtual filesystem locations that we stage files to/from in DataStaging to the places referred to in various places in PosixApplication (especially Argument and Environment, though this also applies to Input, Output and Error; possibly WorkingDirectory too). The problem is that we want to virtualize the location within the real target filesystem(s) that the files are located in, but we do not have any standard substitution scheme; so far, we've only got the ability to specify literal strings which gives either absolute locations (non-portable) or locations that are relative to the working directory (great for the simple cases, but awful for anything complex). In DRMAA, this problem is solved by allowing magical tokens to be put at the start of applicable parameters and defining a substitution scheme for those magical parameters. (This is also the UNICORE approach). That works, but I use the term "magical" with good reason; the values are not easily manipulable with XML processing tools. The other basic alternative, processing with XSLT on the way in, has the problem in that it tends to end up binding things to specific locations far too soon. What I think we need is a way to say that a certain term is to be made relative to a particular virtual filesystem root. As far as I'm concerned, the easiest way of doing this is to put optional attributes on the elements needing this treatment (listed above) that say that the element value needs to be interpreted as being relative to the root of the virtual filesystem named in the attribute, with an absent attribute indicating that the attribute value is to be used literally. Alternatively, we could put a complex type inside the element, but that's much messier IMO. I suggest that the attribute should be called jsdl:filesystem (or rather, whatever we use in DataStaging but I don't remember for sure right now) and we might use it like this fragment: <jsdl:Application> <jsdl:ApplicationName> Gaussian </jsdl:ApplicationName> <jsdl-exec:POSIXApplication> <jsdl-exec:Input jsdl:filesystem="HOME"> input.txt </jsdl-exec:Input> <jsdl-exec:StackSpaceLimit> 8388608 </jsdl-exec:StackSpaceLimit> </jsdl-exec:POSIXApplication> </jsdl:Application> We'd need to decide what predefined virtual filesystems to support, of course. Or have we done that already? :^) Donal.

2 2

OGSA-BES group
by Darren Pulsipher 26 Apr '05

26 Apr '05

For everyone's information the OGSA-BES group is having meetings on Thursday Mornings at the same time as the JSDL meetings on Tuesdays. To join the email list send mail to majordomo(a)ggf.org subscribe ogsa-bes-bof Darren

1 0

JSDL
by Darren Pulsipher 26 Apr '05

26 Apr '05

When: Tuesday, Apr 26, 2005 8:00 AM (MDT) Where: On Line *~*~*~*~*~*~*~*~*~* Here is the web conference for tomorrow -+-----+-----+-----+-----+-----+-----+-----+-----+- Darren Pulsipher has invited you to attend a Microsoft Office Live Meeting. Subject: JSDL Meeting URL: https://www.livemeeting.com/cc/xangoit/join Meeting ID: KC2MK6 Attendee Key: S|h:@K<9K Role: Attendee This Office Live Meeting invitation is a personal invitation meant only for you. It should not be forwarded. If you received this email by mistake or require Live Meeting Assistance, please refer to the Live Meeting Assistance Center at http://r.office.microsoft.com/r/rlidLiveMeeting?p1=7&p2=en_US&p3=LMInfo&p4=…

1 0

RE: JSDL Meeting on Apr 26 2005
by Bill Harris 25 Apr '05

25 Apr '05

Audio for meeting on Tuesday information 1 800-933-3594 at message dial 3000 id is 6444 and confirmed ;-) Talk to you in the morning William Harris Program Architect Oryx follow me - 800 281 6462 Inside cadence 88 202 6462 Cell Phone 801 301 2744 The contact details are: Dial In Numbers: Belgium - 0800-7053 Bezeq - 1-800-940-1786 France - 0800-903109 Germany - 0800-1815765 Hong Kong - 800-968932 Italy - 800874509 Japan - 00531123656 Korea-Tele - 00798-14-800-0512 Netherlands - 0800-0225447 Singapore - 800-1200766 Sweden - 020-7905043 Taiwan - 00801-13-8546 UK - 0800961294 US - 800-933-3594 --> ext 3000 pin: 6444 The telecom time is: wed April 26 this week 07:00 PST 08:00 MST 09:00 CST 10:00 EST 15:00 BST 16:00 CET 23:00 JST William Harris Program Architect Oryx follow me - 800 281 6462 Inside cadence 88 202 6462 Cell Phone 801 301 2744

1 0

Process Topology
by Christopher Smith 21 Apr '05

21 Apr '05

Ok ... here are my thoughts on process topology and what's currently expressible in JSDL. First, I'll list some use cases (they're all parallel jobs): 1. Simple MPI job. Wants 32 processors with 1 processor per resource (in JSDL, a host is a "resource"). 2. OpenMPI job. Wants 32 processors with 8 processors per resource. 3. An OpenMP job. Wants 32 processors. Shared mem of course, so one resource. 4. A "homegrown" master/slave parallel job (say a ligand docking job). Wants 32 processors. No tiling constraints at all. * Note that I'm specifically leaving out the Naregi "coupled simulation" use case (sorry guys), since we determined at the last GGF that it was a case which could be decomposed into multiple JSDL documents. Second ... what is process topology? It provides the user a way to express how resources should be _allocated_ given the characteristics of the job (usually in terms of IO patterns ... e.g. network communication, disk IO channel contention, etc). Thus, it's used when the resource manager is _allocating_ the resources, not when the job is being started/launched. Therefore, none of the elements used to express process topology should be in the POSIXApplication section What we have in JSDL now: ResourceCount (how many "resources" i.e. hosts I want) CPUCount (how many processors _per resource_) TileSize (how many processors to allocate per resource as a unit) ProcessCount (total number of _processes_ that the job will use to execute the job) I will argue that ProcessCount is useless for the purposes of process topology, since a) it isn't about allocation, and b) there isn't enough information to tell me how to start/launch a parallel job. It isn't about allocation since it is irrelevant to the scheduler whether I'll be computing using threads or processes. It isn't useful for launching because it doesn't tell me how to spread the ProcessCount processes given a particular allocated topology. So that leaves the rest of them. TileSize and CPUCount are pretty much the same thing. At least for 80% (or more) of the uses I've seen. The only thing that might cause them to differ is that I could possibly allocate more than one tile on a host. Given that CPUCount is a range and that we could express step values in the range (we can express step values in the range, right?), we don't need TileSize any more. Note: I'm making an assumption here that CPUCount is the number of cpus that I want from the resource, rather than an expression of how many cpus the host needs to have configured. If it is the latter, then we do need TileSize, and replace CPUCount in my examples below with TileSize. So let's see how these map to the use cases. 1. ResourceCount == 32, CPUCount == 1 -> LSF : "-n 32 -R span[ptile=1]" -> PBS : "-l nodes=32:ppn=1" (ppn=1 might be the default) 2. ResourceCount == 4, CPUCount == 8 -> LSF : "-n 32 -R span[ptile=8]" -> PBS : "-l nodes=4:ppn=8" 3. ResourceCount == 1, CPUCount == 32 -> LSF : "-n 32 -R span[hosts=1]" (hosts=1 equivalent to ptile=<-n val>) -> PBS : "-l nodes=1:ppn=32" 4. ResourceCount == 32, CPUCount == 1 -> oops ... it doesn't care about tiling ResourceCount == 1, CPUCount == 32 -> hmm ... artificial constraint ... would suck on a blade cluster ResourceCount == 1-32, CPUCount == 1,32 -> oops again ... I might get a total allocation of 32*32 cpus * there seems to be a gap! If we had a term called "TotalCPUCount" for the entire job, I could do: 4. TotalCPUCount == 32 -> LSF : "-n 32" -> PBS : "not sure how to express" It basically means to grab 32 cpus, regardless of how they are spread. Basically I just need cpus. This is used a whole hell of a lot within our customer base. So ... in summary ... I propose: CPUCount (as is if it's allocated cpus per resource) TileSize (iff CPUCount is an expression of configured cpus in a host) ResourceCount (as is ... hmmm ... maybe the default value needs to change) TotalCPUCount (how many cpus this jobs needs to run in total) -- Chris

3 12

JSDL: Telecon minutes 19 Apr 2005
by Ali Anjomshoaa 21 Apr '05

21 Apr '05

Hi all, the minutes of yesterday's telecon are on GridForge: https://forge.gridforum.org/projects/jsdl-wg/document/Minutes_-_2005-04-19/… Those on the call, you can find your actions in-lined in the minutes. Please ask for help with your actions if you don't have time or need input. Target someone you know is able to help and perhaps has time. Catch you on the call next Tuesday, when we can tick off a few more issues. Cheers, Ali -- ---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali(a)epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------

3 2

Re: [jsdl-wg] Process Topology
by Karl Czajkowski 20 Apr '05

20 Apr '05

On Apr 20, Andreas Savva loaded a tape reading: > Afaik Resource should be 0-n.... Thanks for pointing it out. I'll > confirm it and change the spec. > > When were heterogeneous resource mixes discussed and are there any notes > of that discussion? > I think it came up at GGF-13 and I thought it was 0-n too at the time. I got confused by the 0-1 section title in the draft... I thought we said the general model was a sequence of Resource elements each with its own localally scoped attributes and the resource count was just a shorthand to avoid writing redundant sequences for homogeneous scenarios. If it remains 0-n, I wonder if we should have an enclosing Resources element or something like that to hold the "total" metrics that summarize allocation across all resources? karl -- Karl Czajkowski karlcz(a)univa.com

2 3

Tracker status value proposal - take 2
by Andreas Savva 20 Apr '05

20 Apr '05

Ok, here's a slightly revised proposal for the status values. Again note that this is just for the *specification* tracker and not for the general 'Agenda/Topic' tracker. At the moment the tracker artifact status values are the gridforge default: open, pending, closed. These cannot be changed but new values can be defined. I propose to augment them with the following values: resolved, fixed "open", "pending", "resolved" and "fixed" would be of status class Open "closed" of status class Closed Status value definitions will be clearly documented on the tracker summary page. A typical workflow would be open->pending->resolved->fixed->closed (Michel's email gives a more detailed proposal for these transitions.) The meaning of these values (for the typical path) would be as follows: Open: the issue has been proposed by a member but not yet accepted by the WG. Pending: the issue has been accepted and is still pending resolution by the WG. (Includes 'waiting response from author' mentioned by Donal.) Resolved: The WG has decided and accepted a resolution, but the actions are still pending Fixed: The editor has completed the proposed actions, but the actions are awaiting verification. Both the schema and spec may have to be revised in order to reach this state. Closed: The issue has been resolved and is reflected in the work products of the WG. The completion of the changes must be reported by the primary editor and verified by the secondary editor (or other WG member). Issues that have been deferred or rejected also get closed with an appropriate comment. Issues that have been deferred are moved to the post-v1 tracker. (After checking the tracker settings again, "deferred" or "rejected" should probably be 'resolution' values rather than status values. The resolution field exists but is not enabled at the moment. I propose we first try out the new status values. We can come back and enable the 'resolution' field if we decide we really need it.) Andreas

4 5

[karlcz@univa.com: Re: [jsdl-wg] Process Topology]
by Karl Czajkowski 20 Apr '05

20 Apr '05

oops, hit reply instead of reply-all and missed the JSDL list... -- Karl Czajkowski karlcz(a)univa.com

2 1