- jsdl-wg - lists.ogf.org

JSDL: Telecon minutes
by Ali Anjomshoaa 18 May '05

18 May '05

Hi, you can find yesterday's telecon minutes at: https://forge.gridforum.org/projects/jsdl-wg/document/Minutes_-_2005-05-17/… Andreas will be releasing the next version of the doc and schema by the end of the week. This new version should have the scope of v1.0 complete. Cheers and take care, Ali -- ---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali(a)epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------

1 0

Proposal for Resource section
by Andrew Stephen McGough 17 May '05

17 May '05

All, Here is the idea Donal and I had during the JSDL telecon. We make the Resource element into a Resources element, this discribes the overall requirement for the job. Some elements are prepended with Total others with Individual (and some both). This then deals with the majority of job submissions - if you wish to describe different Resource elements then this can be added in as an extension. Outlined are the changes below: <Resources> <CandidateHosts ... />? <CPUArchitecture .../>? vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv <IndividualCPUSpeed .../>? <OverallCPUTime .../>? <TotalCPUCount .../>? <IndividualCPUCount .../>? <TotalResourceCount .../>? <IndividualPhysicalMemory .../>? <TotalPhysicalMemory .../>? <IndividualVirtualMemory .../>? <TotalNetworkBandwidth .../>? <IndividualNetworkBandwidth .../>? <TotalDiskSpace../>? <IndividualDiskSpace .../>? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ <FileSystem .../>* <ExlusiveExecution .../>? <OperatingSystem .../>? <xsd:any##other>* </Resources>* steve.. (and Donal) -- -- ------------------------------------------------------------------------ Dr A. Stephen McGough http://www.doc.ic.ac.uk/~asm ------------------------------------------------------------------------ Technical Coordinator, London e-Science Centre, Imperial College London, Department of Computing, 180 Queen's Gate, London SW7 2BZ, UK tel: +44 (0)207-594-8409 fax: +44 (0)207-581-8024 ------------------------------------------------------------------------ Assistant Warden, West Wing, Beit Hall, Imperial College, Prince Consort Road, London, SW7 2BB tel: +44 (0)207-594-9910 ------------------------------------------------------------------------

2 2

JSDL: MeetingPhone.co.uk Conference Call Confirmation
by Ali Anjomshoaa 17 May '05

17 May '05

Folks, details below.... Cheers, Ali Conference Details Date: 17 May 2005 Start Time: 3:15 p.m. British Summer Time End Time: 5:10 p.m. British Summer Time Primary Dial-in Number: 0871.711.1515 (UK 10p National Rate) * +44.870.088.5700 (For Callers Outside of the U.K.) * 01805.004.101 (For Callers in Germany) Participant Access Code: 17052005 Reminder about access codes: The Organiser Access Code is necessary for enhanced conference controls. If you do not wish to use the enhanced controls, you may disregard the Organiser Access Code and use the Participant Access Code instead. Touch-Tone Quick Reference for Conference Call 0 Exit the Conference 4 Access the Help Menu 6 Toggle Mute/Unmute 8 Time Remaining in the Conference # Hear the Number of Parties in the Conference See you there!

1 1

summary of topology thread
by Karl Czajkowski 13 May '05

13 May '05

As promised on the telecon, I try to summarize the "topology thread." The basic status is that Chris and I seem to be in agreement about a range of resource constraints that need expression and there is a minor open question about how (if at all) the binding of application to resource is expressed. I have talked to Chris and others from Platform a lot over the years, so I guess it is not clear whether we've made progress explaining this position to others. :-/ PROPOSED SET OF CONCEPTS What Chris and I think is that there are three critical concepts that need expression and they all relate to what allocation the job will receive. We think these should be expressed in the resource section: A. Total Number of CPUs for job B. Number of CPUs per resource (e.g. per LSF host) C. Total Number of resources I think both (B) and (C) are tiling constraints and need to default to "unconstrained" if left out of the JSDL doc. It is important to note that by expressing all of (A), (B), and (C) one can overconstrain the selection, e.g. describe redundant things or impossible things. We think this is OK and maybe even necessary in order to allow expression of the important things. All three values should be range-value for generalized constraints w/ step values, limits, etc. Another minor point is whether we want to express shared/exclusive access to resources. Saying "4 CPUs per resource" DOES NOT mean that all allocated hosts will have 4 installed CPUs. It means the job is granted the right to use 4 of the CPUs, but there could be more which are kept idle or allocated to other jobs. REPRESENTATION OF CONCEPTS The total value (A) is the difficult one, because it applies across all Resource elements but does not really belong in the Application section in our view. I suggest that a wrapping layer could be placed around Resource: <jsdl:Resources> <jsdl:TotalCPUs>jsdl:rangeValue</jsdl:TotalCPUs> ? ... other totals for RAM, VM, etc. ... <jsdl:Resource> <jsdl:Count>jsdl:rangeValue</jsdl:Count> ? <jsdl:CPUs>jsdl:rangeValue</jsdl:CPUs> ? ... other per-resource constraints ... </jsdl:Resource> + </jsdl:Resources> Make jsdl:TotalCPUs default to "<exact>1</exact>". Make jsdl:Count and jsdl:CPUs default to unconstrained. The remaining question, again, is how to represent the binding of application to resource. This is where the GRAM single/multi/mpi/condor stuff fits in to state different launch techniques. It might be appropriate to say the default is "site specific" and a reasonable assumption is that a real resource topology is selected by the scheduler and somehow communicated to the job in the form of a resources file, environment setting, or actual process set startup. One can imagine arbitrarily strange and complex scenarios here for the interesting cases. USE CASES TO ILLUSTRATE CONCEPTS Chris made a nice concise statement of use cases that need to be addressed, and I think the implication is that JSDL examples should use these cases rather than existing confusing ones: 1. Simple MPI job. Wants 32 processors with 1 processor per resource (in JSDL, a host is a "resource"). <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:CPUs><exact>1</exact></jsdl:CPUs> ... </jsdl:Resource> </jsdl:Resources> <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:Count><exact>32</exact></jsdl:Count> ... </jsdl:Resource> </jsdl:Resources> <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:Count><exact>32</exact></jsdl:Count> <jsdl:CPUs><exact>1</exact></jsdl:CPUs> ... </jsdl:Resource> </jsdl:Resources> All three of these lead to exactly the same set of possible allocations. I will not show equivalent permutations for the rest of the examples but this gives the basic idea. 2. OpenMPI job. Wants 32 processors with 8 processors per resource. <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:CPUs><exact>8</exact></jsdl:CPUs> ... </jsdl:Resource> </jsdl:Resources> 3. An OpenMP job. Wants 32 processors. Shared mem of course, so one resource. <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> <jsdl:Count><exact>1</exact></jsdl:Count> ... </jsdl:Resource> </jsdl:Resources> 4. A "homegrown" master/slave parallel job (say a ligand docking job). Wants 32 processors. No tiling constraints at all. <jsdl:Resources> <jsdl:TotalCPUs><exact>32</exact></jsdl:TotalCPUs> ... <jsdl:Resource> ... </jsdl:Resource> </jsdl:Resources> * Note that I'm specifically leaving out the Naregi "coupled simulation" use case (sorry guys), since we determined at the last GGF that it was a case which could be decomposed into multiple JSDL documents. Of course, these use cases leave out many interesting "loose" constraints where range-value is used to allow many possibly heterogeneous solutions to the constraint system of the JSDL doc. I would add that I think the "coupled simulation" use case can be addressed by the resource constraints we are proposing but it is in the "binding of application to resource" where multiple JSDL docs are required. For example, we can select heterogeneous resources if the single application knows how to run each different "simulation" on subsets of the allocation without expressing that binding in JSDL: <jsdl:Resources> ... <jsdl:Resource> <jsdl:Count><exact>8</exact></jsdl:Count> <jsdl:CPUs><exact>2</exact></jsdl:CPUs> ... </jsdl:Resource> <jsdl:Resource> <jsdl:Count><exact>16</exact></jsdl:Count> <jsdl:Count><exact>1</exact></jsdl:Count> ... </jsdl:Resource> </jsdl:Resources> If we wanted to address such complex scenarios more elegantly, we could allow more than one Resources element or even allow Resources elements to recursively contain Resources elements in order to express logical assemblies of resources which scope the "total" constraints expressed within them as a summation of the contained assemblies or "leaf" resource elements. I do not know if anyone cares about this sort of hierarchical resource map. karl -- Karl Czajkowski karlcz(a)univa.com

3 7

JSDL Schema Doc
by Andrew Stephen McGough 12 May '05

12 May '05

All, I'm sat on a train so a perfect time to comment on the document... 1) Figure 1 is great now - though it isn't referred to in the text. Perhapse at the end of the paragraph beforehand: "Figure 1 illistrates places where a JSDL document may be used within a Grid environment. It should be noted that the JSDL document should not be limited to just these situations as many others may exist.". 2) First paragraph of section 3: Last scentence: Many of these languages are currently undefined though _they_ are required for the submission of jobs. 3) Page 8: Job Policy Thus -> This. 4) Page 8: Workflow: Incomplete scentence: Many other groups within (and without) the GGF. -> Many other groups witin (and without) the GGF are already working within this area. 5) Section 3.1 has an [add more text here] element. 6) Page 24: Example of a MoutSource The filesystem "HOME" must have at most 10GB capacity. The execution system is free to choose an appropriate MountPoint since no value is provided. This should mount the NFS export sputnick.ggf.org:/home/steve <jsdl:FileSystem name="jsdl:HOME" > <jsdl:Description> Steve's home </jsdl:Description> <jsdl:MountSource>nfs:sputnick.ggf.org/home/steve</jsdl:MountSource> <DiskSpace> <UpperBoundedRange>10737418240.0</UpperBoundedRange> </DiskSpace> <jsdl:FileSystemType>jsdl:normal</jsdl:FileSystemType> </jsdl:FileSystem> 7) my email is now just asm(a)doc.ic.ac.uk That's probably enuf for now. steve.. -- -- ------------------------------------------------------------------------ Dr A. Stephen McGough http://www.doc.ic.ac.uk/~asm ------------------------------------------------------------------------ Technical Coordinator, London e-Science Centre, Imperial College London, Department of Computing, 180 Queen's Gate, London SW7 2BZ, UK tel: +44 (0)207-594-8409 fax: +44 (0)207-581-8024 ------------------------------------------------------------------------ Assistant Warden, West Wing, Beit Hall, Imperial College, Prince Consort Road, London, SW7 2BB tel: +44 (0)207-594-9910 ------------------------------------------------------------------------

3 2

CREAM JDL vs JSDL
by Massimo Sgaravatto - INFN Padova 12 May '05

12 May '05

Dear all In the context of the Compute Resource Management Interfaces (CRM) initiative (the same mentioned by Oxana), the Job Description Language (JDL) used by the CREAM (Computing Resource Execution AND Management) service has been compared with JSDL. The CREAM JDL is a subset of the EGEE WMS JDL, that is the language used to specify characteristics and requirements for jobs submitted to the EGEE Workload Management System. This JDL is a high-level, user-oriented language based con Condor Classads for describing jobs and complex requests (aggregates of jobs). It is a fully extensible language, hence the user is allowed to use whatever attribute for the description of a request. However, only a certain set of attributes, referred as "supported attributes", is taken into account. The result of this evaluation is available at: http://www.pd.infn.it/grid/jra1/CREAMJDL-vs-JSDL.pdf A presentation summarizing these results is also available at: http://www.pd.infn.it/~sgaravat/Grid/CREAMJDL-vs-JSDL-EGEE3.pdf http://www.pd.infn.it/~sgaravat/Grid/CREAMJDL-vs-JSDL-EGEE3.ppt In short we found that for many functionality for which specific CREAM JDL attributes exist there aren't "default" JSDL directives that could be used. For most of them this is not surprising since they refer to features/functionality very specific to our architecture. In this case I guess JSDL extensions could be used. On the other hand (and here I agree with Oxana) there are some issues that in my opinion could/should probably be addressed by the default JSDL directives. In particular (and, again, I agree with Oxana): - the specification of workflows (in our case DAGs, job collections, parametric jobs) - the support for matchmaking, i.e. allowing specifying requirements (and preferences) on the target resource where the job has to be executed. Thanks for your attention Cheers, Massimo \\\|/// \\ ~ ~ // (/ @ @ /) -------oOOo-(_)-oOOo---------------------------------- Massimo Sgaravatto INFN Sezione di Padova Via Marzolo, 8 35131 Padova - Italy Tel: ++39 0498277047 Fax: ++39 0498277102 oooO E-mail: massimo.sgaravatto(a)pd.infn.it ( ) Oooo Home page: http://www.pd.infn.it/~sgaravat --------\ (----( )---------------------------------- \_) ) / (_/

2 1

JSDL vs NorduGrid xRSL: a comparison (fwd)
by Ali Anjomshoaa 11 May '05

11 May '05

Forwarding bounced mail on behalf of Oxana Smirnova, NorduGrid. Ali -- ---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali(a)epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 ------------------------------------------------------------- ---------- Forwarded message ---------- Date: Mon, 09 May 2005 14:52:43 +0200 From: Oxana Smirnova <oxana.smirnova(a)hep.lu.se> To: jsdl-wg(a)gridforum.org, egee-project-crm <egee-project-crm(a)cern.ch>, NorduGrid Discussion <nordugrid-discuss(a)nordugrid.org> Subject: JSDL vs NorduGrid xRSL: a comparison Dear all, as agreed by the Compute Resource Management Interfaces (CRM) initiative, JSDL has been evaluated by the participating Grid system experts against job description languages presently in use. NorduGrid's ARC uses extended Globus RSL (xRSL), which can be quite easily mapped to the POSIX Application of JSDL. However, there are several elements that are missing from JSDL but have been proven to be essential for proper Grid job description and execution - at least from the NorduGrid perspective. JSDL appears to be a proper framework for a more advanced Grid job description, provided it can be extended to: a) accomodate more elements (most notably, Run-time Environment for an application) b) enable multiple job definitions in a single "script"/file c) allow for alternative parameter requests (e.g., alternative OS, environment, host etc) A detailed comparison and related notes can be downloaded at http://www.nordugrid.org/documents/jsdl-vs-xrsl-revised2.sxw Hopefully, this can be taken as an input for further JSDL development. NorduGrid so far did not participate in JSDL-WG activities, but can contribute in future, if the WG members find it useful. Best regards, Oxana Smirnova, for the NorduGrid collaboration

4 5

JSDL: Telecon minutes and actions
by Ali Anjomshoaa 11 May '05

11 May '05

Hi all, the minutes from yesterday's telecon are attached. You can also find them at: https://forge.gridforum.org/projects/jsdl-wg/document/Minutes_-_2005-05-10/… Please take note of your Actions from these minutes. We're very close to closing the issues for v1.0, after which an editorial polish is due. Version 1.0 is due for submission to the GGF Editor for AD review on Wed. 25th May - 2 weeks from today. The plan is to freeze the contents next Wed. 18th. Cheers and take care, Ali -- ---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali(a)epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------

1 0

JSDL Meeting
by Darren Pulsipher 10 May '05

10 May '05

When: Tuesday, May 10, 2005 7:00 AM (MDT) Where: Online *~*~*~*~*~*~*~*~*~* -+-----+-----+-----+-----+-----+-----+-----+-----+- Darren Pulsipher has invited you to attend a Microsoft Office Live Meeting. Subject: JSDL Meeting Meeting URL: https://www.livemeeting.com/cc/xango/join Meeting ID: WXCXR7 Attendee Key: KF~44D{'b Role: Attendee This Office Live Meeting invitation is a personal invitation meant only for you. It should not be forwarded. If you received this email by mistake or require Live Meeting Assistance, please refer to the Live Meeting Assistance Center at http://r.office.microsoft.com/r/rlidLiveMeeting?p1=7&p2=en_US&p3=LMInfo&p4=…

1 0

Fwd: JSDL action point 17: Extending JSDL
by Michel Drescher 09 May '05

09 May '05

Folks, since this is something that is required for JSDL v1 and I unfortunately did not get Steve's comments on this yet, I decided to give the JSDL community the opportunity to comment on this before tomorrow's telecon. Cheers, Michel Begin forwarded message: > From: Michel Drescher <Michel.Drescher(a)uk.fujitsu.com> > Date: 3 May 2005 17:45:49 BST > To: Andrew Stephen McGough <asm(a)doc.ic.ac.uk> > Subject: Re: JSDL action point 17: Extending JSDL > > Hi Steve, > > thx for the review. I now come up with the following, having added a > section for jsdl:other and an example for attribute extension: > > Extending JSDL > ============== > JSDL provides for the overall structure to define jobs. To extend JSDL > to best fit to the required needs, JSDL provides two mechanisms for > extension: Extending using attributes and extending using elements. In > general using extension mechanisms will break interoptability of JSDL > consumers and as such these mechanisms should be used spareingly and > only where need. It is recommended that people interested in extending > the functionality of JSDL should first verify that no other group has > provided an extension which meets their requirements. > > Attribute based extensions > -------------------------- > Every JSDL element allows for additional attributes, as many as > needed, provided these attributes have a namespace other than the > normative namespaces JSDL defines. > > Example: > This example introduces an order that has to be followed when staging > in files (shortened for brevity): > > <?xml version="1.0" encoding="UTF-8"?> > <jsdl:JobDefinition xmlns="http://www.example.org/" > xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/04/jsdl" > xmlns:o="http://www.example.org/order-of-execution"> > <jsdl:JobDescription> > <jsdl:DataStaging o:order="1"> > <jsdl:FileName>foo</jsdl:FileName> > <jsdl:CreationFlag>overwrite</jsdl:CreationFlag> > > <jsdl:Source><jsdl:URL>http://www.nowhere.com/foo-file</jsdl:URL></ > jsdl:Source> > </jsdl:DataStaging> > <jsdl:DataStaging o:order="2"> > <jsdl:FileName>bar</jsdl:FileName> > <jsdl:CreationFlag>overwrite</jsdl:CreationFlag> > > <jsdl:Source><jsdl:URL>http:://www.nowhere.com/bar-file</jsdl:URL></ > jsdl:Source> > </jsdl:DataStaging> > </jsdl:JobDescription> > </jsdl:JobDefinition> > > > > Element based extensions > ------------------------ > Where applicable within the overall structure of the document, JSDL > allows for additional elements that are not normatively defined by > JSDL. As with attribute extension, these elements must assume a > namespace different than any namespace JSDL normatively defines. > > Example (shortened for brevity): > The following example extends JSDL by introducing reservations: > <?xml version="1.0" encoding="UTF-8"?> > <jsdl:JobDefinition xmlns="http://www.example.org" > xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/04/jsdl"> > <jsdl:JobDescription> > <jsdl:Resource> > <jsdl:CPUCount>  > <jsdl:Exact>1.0</jsdl:Exact> > </jsdl:CPUCount> > <jsdl:DiskSpace>  > > <jsdl:LowerBoundedRange>1073741824.0</jsdl:LowerBoundedRange> > </jsdl:DiskSpace> > <ug:Reservation > > xmlns:ug="http://schemas.unigrids.org/2005/05/reservation"> > <ug:Ticket>h933fsolenri900wnmd90mm34</ug:Ticket> > </ug:Reservation> > </jsdl:Resource> > </jsdl:JobDescription> > </jsdl:JobDefinition> > > Semantics of jsdl:other > ----------------------- > JSDL defines enumerations of values for processor architectures and > operating system. Both enumerations are not assumed to be exhaustively > complete. These enumerations are embedded in wrapper types to allow > for extension, "jsdl:CPUArchitectureType" and > "jsdl:OperatingSystemTypeType", respectively. > > To extend the enumerations, the special keyword "other" MUST be used > for the values of the elements "jsdl:CPUArchitectureName" and > "jsdl:OperatingSystemName", respectiveely, and the element extension > pattern as described above MUST be used. > > Example: > Given there exist a CPU that natively executes Java Bytecode named > "JavaCPU", this CPU would have to be specified as follows in the > resource section of a JSDL instance document: > ... > <Resource> > <CPUArchitecture> > <CPUArchitectureName>other</CPUArchitectureName> > <ex:NewCPUTypes > xmlns:ex="http://www.example.org/NewCPUTypes">JavaCPU</ex:NewCPUTypes> > </CPUArchitecture> > </Resource> >

1 0