May 2005 - jsdl-wg - lists.ogf.org

JSDL: Telecon today
by Ali Anjomshoaa 31 May '05

31 May '05

Hi All, today's telecon details are below. Cheers, Ali -- ---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali(a)epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 ------------------------------------------------------------- -+-----+-----+-----+-----+-----+-----+-----+-----+- Darren Pulsipher has invited you to attend a Microsoft Office Live Meeting. Subject: JSDL Meeting Meeting URL: https://www.livemeeting.com/cc/xangoit/join Meeting ID: JSDL0531 Attendee Key: tzj6#wZ(t Role: Attendee Audio Conferencing (Toll): +1 (770) 765-9436 Audio Conferencing (Toll-free): +1 (886) 765-2460 Participant Code: 8018168214 This Office Live Meeting invitation is a personal invitation meant only for you. It should not be forwarded. If you received this email by mistake or require Live Meeting Assistance, please refer to the Live Meeting Assistance Center at http://r.office.microsoft.com/r/rlidLiveMeeting?p1=7&p2=en_US&p3=LMInfo&p4=…

1 1

Canceled:JSDL
by Darren Pulsipher 31 May '05

31 May '05

When: Tuesday, May 31, 2005 8:00 AM (MDT) Where: On Line *~*~*~*~*~*~*~*~*~* Here is the web conference for tomorrow -+-----+-----+-----+-----+-----+-----+-----+-----+- Darren Pulsipher has invited you to attend a Microsoft Office Live Meeting. Subject: JSDL Meeting URL: https://www.livemeeting.com/cc/xangoit/join Meeting ID: KC2MK6 Attendee Key: S|h:@K<9K Role: Attendee This Office Live Meeting invitation is a personal invitation meant only for you. It should not be forwarded. If you received this email by mistake or require Live Meeting Assistance, please refer to the Live Meeting Assistance Center at http://r.office.microsoft.com/r/rlidLiveMeeting?p1=7&p2=en_US&p3=LMInfo&p4=…

1 0

JSDL Meeting
by Darren Pulsipher 31 May '05

31 May '05

When: Tuesday, May 31, 2005 8:00 AM (MDT) Where: On Line *~*~*~*~*~*~*~*~*~* -+-----+-----+-----+-----+-----+-----+-----+-----+- Darren Pulsipher has invited you to attend a Microsoft Office Live Meeting. Subject: JSDL Meeting Meeting URL: https://www.livemeeting.com/cc/xangoit/join Meeting ID: JSDL0531 Attendee Key: tzj6#wZ(t Role: Attendee Audio Conferencing (Toll): +1 (770) 765-9436 Audio Conferencing (Toll-free): +1 (886) 765-2460 Participant Code: 8018168214 This Office Live Meeting invitation is a personal invitation meant only for you. It should not be forwarded. If you received this email by mistake or require Live Meeting Assistance, please refer to the Live Meeting Assistance Center at http://r.office.microsoft.com/r/rlidLiveMeeting?p1=7&p2=en_US&p3=LMInfo&p4=…

1 0

JSDL specification and schema snapshots (draft 19)
by Andreas Savva 27 May '05

27 May '05

I have uploaded a new snapshot of the JSDL specification and schemas to gridforge. There are too many changes to mention. (Some were the result of side-discussions in parallel with the OGSA f2f. Thanks Chris, Darren, Donal, Michel and Steve.) The MS word file is change tracked. The pdf file is 'clean' and is intended for submission to meet the GGF14 deadline. We'll be moving to clean up and hopefully submit to the GGF Editor over the next 1-2 weeks. MS-Word file: https://forge.gridforum.org/projects/jsdl-wg/document/draft-ggf-jsdl-spec/e… Schemas: https://forge.gridforum.org/projects/jsdl-wg/document/jsdl.xsd/en/13 https://forge.gridforum.org/projects/jsdl-wg/document/jsdl-posix.xsd/en/3 PDF file: https://forge.gridforum.org/projects/jsdl-wg/document/draft-ggf-jsdl-spec-f… -- Andreas Savva

3 2

JSDL: No telecon this week
by Ali Anjomshoaa 23 May '05

23 May '05

Hi everyone, just to let you know that there will not be a telecon tomorrow. We will resume normal operations next week. Cheers, Ali -- ---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali(a)epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------

1 0

DataStaging concerns
by Peter G.Lane 21 May '05

21 May '05

Hi all, Forgive me if I'm reiterating on a topic. I've only be reading up on JSDL since yesterday. I have a few concerns about the DataStaging section. Primarily, I'm wondering if it really makes sense to have it as part of the core schema. I think it would be better to have extensions like POSIXApplication for more specific DRM configurations. Here are some of my thoughts: 1) There's still controversy over whether staging should or should not be integrated into a DRM. As far as I can tell, for example, the BES doesn't have any plans to implement staging. DRMMA makes this optional. If BES ends up using JSDL, wouldn't this be a violation of the spec which requires each element to be supported in some way? 2) There's no distinction between a stage-in or stage-out flavor of the staging directives. I guess it's up to the service to decipher this so that it can perform the staging at the appropriate point in the life cycle. 3) I don't particularly like that the DataStaging sections include an option to remove the file at the end of the job. If I'm staging out data then this doesn't make a whole lot of sense. I'd much prefer a separate section which explicitly lists all the files that are to be removed from the submission machine after the job has been completed. This would also cover the case of data that is created rather than staged in but still needs to be removed after job completion. 4) Based on the current GRAM incarnation, it would be nice to let RFT's transfer request description extend a base staging schema and then use that in the JSDL document rather than adding a bunch of extensions to DataStaging. This is similar to how I'd want to go about using POSIXApplication. Thanks in advance for any responses... Peter

5 9

JSDL: Telecon minutes
by Ali Anjomshoaa 18 May '05

18 May '05

Hi, you can find yesterday's telecon minutes at: https://forge.gridforum.org/projects/jsdl-wg/document/Minutes_-_2005-05-17/… Andreas will be releasing the next version of the doc and schema by the end of the week. This new version should have the scope of v1.0 complete. Cheers and take care, Ali -- ---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali(a)epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------

1 0

Proposal for Resource section
by Andrew Stephen McGough 17 May '05

17 May '05

All, Here is the idea Donal and I had during the JSDL telecon. We make the Resource element into a Resources element, this discribes the overall requirement for the job. Some elements are prepended with Total others with Individual (and some both). This then deals with the majority of job submissions - if you wish to describe different Resource elements then this can be added in as an extension. Outlined are the changes below: <Resources> <CandidateHosts ... />? <CPUArchitecture .../>? vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv <IndividualCPUSpeed .../>? <OverallCPUTime .../>? <TotalCPUCount .../>? <IndividualCPUCount .../>? <TotalResourceCount .../>? <IndividualPhysicalMemory .../>? <TotalPhysicalMemory .../>? <IndividualVirtualMemory .../>? <TotalNetworkBandwidth .../>? <IndividualNetworkBandwidth .../>? <TotalDiskSpace../>? <IndividualDiskSpace .../>? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ <FileSystem .../>* <ExlusiveExecution .../>? <OperatingSystem .../>? <xsd:any##other>* </Resources>* steve.. (and Donal) -- -- ------------------------------------------------------------------------ Dr A. Stephen McGough http://www.doc.ic.ac.uk/~asm ------------------------------------------------------------------------ Technical Coordinator, London e-Science Centre, Imperial College London, Department of Computing, 180 Queen's Gate, London SW7 2BZ, UK tel: +44 (0)207-594-8409 fax: +44 (0)207-581-8024 ------------------------------------------------------------------------ Assistant Warden, West Wing, Beit Hall, Imperial College, Prince Consort Road, London, SW7 2BB tel: +44 (0)207-594-9910 ------------------------------------------------------------------------

2 2

JSDL: MeetingPhone.co.uk Conference Call Confirmation
by Ali Anjomshoaa 17 May '05

17 May '05

Folks, details below.... Cheers, Ali Conference Details Date: 17 May 2005 Start Time: 3:15 p.m. British Summer Time End Time: 5:10 p.m. British Summer Time Primary Dial-in Number: 0871.711.1515 (UK 10p National Rate) * +44.870.088.5700 (For Callers Outside of the U.K.) * 01805.004.101 (For Callers in Germany) Participant Access Code: 17052005 Reminder about access codes: The Organiser Access Code is necessary for enhanced conference controls. If you do not wish to use the enhanced controls, you may disregard the Organiser Access Code and use the Participant Access Code instead. Touch-Tone Quick Reference for Conference Call 0 Exit the Conference 4 Access the Help Menu 6 Toggle Mute/Unmute 8 Time Remaining in the Conference # Hear the Number of Parties in the Conference See you there!

1 1

summary of topology thread
by Karl Czajkowski 13 May '05

13 May '05

As promised on the telecon, I try to summarize the "topology thread." The basic status is that Chris and I seem to be in agreement about a range of resource constraints that need expression and there is a minor open question about how (if at all) the binding of application to resource is expressed. I have talked to Chris and others from Platform a lot over the years, so I guess it is not clear whether we've made progress explaining this position to others. :-/ PROPOSED SET OF CONCEPTS What Chris and I think is that there are three critical concepts that need expression and they all relate to what allocation the job will receive. We think these should be expressed in the resource section: A. Total Number of CPUs for job B. Number of CPUs per resource (e.g. per LSF host) C. Total Number of resources I think both (B) and (C) are tiling constraints and need to default to "unconstrained" if left out of the JSDL doc. It is important to note that by expressing all of (A), (B), and (C) one can overconstrain the selection, e.g. describe redundant things or impossible things. We think this is OK and maybe even necessary in order to allow expression of the important things. All three values should be range-value for generalized constraints w/ step values, limits, etc. Another minor point is whether we want to express shared/exclusive access to resources. Saying "4 CPUs per resource" DOES NOT mean that all allocated hosts will have 4 installed CPUs. It means the job is granted the right to use 4 of the CPUs, but there could be more which are kept idle or allocated to other jobs. REPRESENTATION OF CONCEPTS The total value (A) is the difficult one, because it applies across all Resource elements but does not really belong in the Application section in our view. I suggest that a wrapping layer could be placed around Resource: <jsdl:Resources> <jsdl:TotalCPUs>jsdl:rangeValue</jsdl:TotalCPUs> ? ... other totals for RAM, VM, etc. ... <jsdl:Resource> <jsdl:Count>jsdl:rangeValue</jsdl:Count> ? <jsdl:CPUs>jsdl:rangeValue</jsdl:CPUs> ? ... other per-resource constraints ... </jsdl:Resource> + </jsdl:Resources> Make jsdl:TotalCPUs default to "<exact>1</exact>". Make jsdl:Count and jsdl:CPUs default to unconstrained. The remaining question, again, is how to represent the binding of application to resource. This is where the GRAM single/multi/mpi/condor stuff fits in to state different launch techniques. It might be appropriate to say the default is "site specific" and a reasonable assumption is that a real resource topology is selected by the scheduler and somehow communicated to the job in the form of a resources file, environment setting, or actual process set startup. One can imagine arbitrarily strange and complex scenarios here for the interesting cases. USE CASES TO ILLUSTRATE CONCEPTS Chris made a nice concise statement of use cases that need to be addressed, and I think the implication is that JSDL examples should use these cases rather than existing confusing ones: 1. Simple MPI job. Wants 32 processors with 1 processor per resource (in JSDL, a host is a "resource"). <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:CPUs><exact>1</exact></jsdl:CPUs> ... </jsdl:Resource> </jsdl:Resources> <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:Count><exact>32</exact></jsdl:Count> ... </jsdl:Resource> </jsdl:Resources> <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:Count><exact>32</exact></jsdl:Count> <jsdl:CPUs><exact>1</exact></jsdl:CPUs> ... </jsdl:Resource> </jsdl:Resources> All three of these lead to exactly the same set of possible allocations. I will not show equivalent permutations for the rest of the examples but this gives the basic idea. 2. OpenMPI job. Wants 32 processors with 8 processors per resource. <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:CPUs><exact>8</exact></jsdl:CPUs> ... </jsdl:Resource> </jsdl:Resources> 3. An OpenMP job. Wants 32 processors. Shared mem of course, so one resource. <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:Count><exact>1</exact></jsdl:Count> ... </jsdl:Resource> </jsdl:Resources> 4. A "homegrown" master/slave parallel job (say a ligand docking job). Wants 32 processors. No tiling constraints at all. <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> ... </jsdl:Resource> </jsdl:Resources> * Note that I'm specifically leaving out the Naregi "coupled simulation" use case (sorry guys), since we determined at the last GGF that it was a case which could be decomposed into multiple JSDL documents. Of course, these use cases leave out many interesting "loose" constraints where range-value is used to allow many possibly heterogeneous solutions to the constraint system of the JSDL doc. I would add that I think the "coupled simulation" use case can be addressed by the resource constraints we are proposing but it is in the "binding of application to resource" where multiple JSDL docs are required. For example, we can select heterogeneous resources if the single application knows how to run each different "simulation" on subsets of the allocation without expressing that binding in JSDL: <jsdl:Resources> ... <jsdl:Resource> <jsdl:Count><exact>8</exact></jsdl:Count> <jsdl:CPUs><exact>2</exact></jsdl:CPUs> ... </jsdl:Resource> <jsdl:Resource> <jsdl:Count><exact>16</exact></jsdl:Count> <jsdl:Count><exact>1</exact></jsdl:Count> ... </jsdl:Resource> </jsdl:Resources> If we wanted to address such complex scenarios more elegantly, we could allow more than one Resources element or even allow Resources elements to recursively contain Resources elements in order to express logical assemblies of resources which scope the "total" constraints expressed within them as a summation of the contained assemblies or "leaf" resource elements. I do not know if anyone cares about this sort of hierarchical resource map. karl -- Karl Czajkowski karlcz(a)univa.com

3 7