October 2005 - saga-rg - lists.ogf.org

Minutes
by Thilo Kielmann 05 Oct '05

05 Oct '05

All, attached please find the minutes for the SAGA-RG meeting of 04/10/05. All errors and typos are solely mine... Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/

1 0

Use Case(s) feedback....
by Shantenu Jha 05 Oct '05

05 Oct '05

Appended below is feedback on the API from six Use Case authors, (and some SAGA counter comments -- some of which were discussed in the SAGA-RG session on Mon 03 Oct). Use case authors were asked the following: ====== Subject: SAGA API specification feedback Hi, This mail reaches you because you are listed as contact for a Use Case [1] submitted to the SAGA Research Group [2] at GGF [3]. The SAGA API v0.2 specification which is based on the submitted use cases (and other input) has stablized sufficiently over the past few months and is now rapidly converging toward a GGF submission. We would be grateful if you as potenital 'clients' of the SAGA API could review the current draft, verify that it indeed serves your use cases. Please also, let us know your frank opinion if you think the current spec satisfies the all important "S" -- Simple -- in SAGA. Any feedback on what you might like to see done differently will also be very useful. Also, we would like to invite you to a dedicated session at GGF15 [4] (exact time and date to be announced) to discuss the mappings of the API to the use cases. The Use Case collection (including your use case) can be found at [5]. The SAGA API draft is available at [6], a short version of the spec containing the API only can be found at [7]. For general information on the SAGA group, please check [8] and [9]. With best regards, SAGA-RG. [1] http://wiki.cct.lsu.edu/saga/space/start/use-cases.pdf [2] http://forge.gridforum.org/projects/saga-rg/ [3] http://www.ggf.org/ [4] http://www.ggf.org/ggf_events_ggf15.htm [5] http://wiki.cct.lsu.edu/saga/space/start/use-cases.pdf [6] http://wiki.cct.lsu.edu/saga/space/start/strawman-api-v0.2.pdf [7] http://wiki.cct.lsu.edu/saga/space/start/strawman-api-v0.2.short.pdf [8] http://forge.gridforum.org/projects/saga-rg/ [9] http://wiki.cct.lsu.edu/saga/space/start ==== And here is are their responses: Grid SuperScalar: ----------------- The SAGA API version 0.2 is useful for our requirements but we find some things missing. We detail more each part of the API: - Session and Context: We don't have any special requirements in this area. - Error: No information given in the specification. - Task: As it is supposed to be the asyncronous version of each SAGA API method, we think that this may be also useful for us in order to do asyncronous job submission, like the one achieved with globus_gram_client_register_job_request in contrast with globus_gram_client_job_request. - Attributes and Name spaces: No special comments. - Files and Logical Files: This two interfaces are very good if you want to access to a remote file in the first case, and to work with a replica location system in the second case, but nothing is provided in order to copy files from one machine to another if you don't want to use a replica system. In our use case, our run-time is aware of the location of files, so an easy mechanism in order to copy files between machines must be provided. This is not easy to do as the API is now. A possible solution is to include copy, move or erase methods for files in the File API. <<< Comment Andre: -------------- I think there is a misunderstanding. The file packages is inheriting from the NameSpace package. That is, a saga::file implements the saga::name_space_entry interface, and a saga::directory implements the saga::name_space_directory interface. All operations which are agnostic to content of the file (such as create, copy, move, open, rename, list, ...) are defined in the namespace interface. That interface is also inherited by the logical_file package, so the same methods are available there. Hence, the logical_file and the file packages only have those methods defined, which distinguish them from simple name spaces. >>> - Jobs: It covers our needs in order to describe a job (also in terms of adding restrictions to a job) and we think the job state diagram is complete. The only thing we find is missing is a call for waiting till a notification arrives (whatever the notification is). We submit several jobs at the same time, and we need to receive notifications of the states of those jobs in order to take actions in our run-time. So we follow a notification model instead of a polling model (we wait for notifications for arrive, instead of polling for state changes from our run-time). In our opinion the API would be more complete if it includes both models for job state control (polling and notification model), so this gives more freedom to the API user. - Streams: This API also is useful for us, in order to achieve communication between the workers and the master in an easy way to exchange a reduced amount of information. ====== Pascal Kleijer: --------------- Here are some pointers on what can be done for the future revisions. - Naming. The different APIs do not respect all the same naming pattern. Some like files are a direct UNIX style command line translation (i.e. "ls" should be "list"). I would recommend using a uniform method, attributes and constant naming. If you use OO design then stick with OO paradigm. Use full naming instead of acronyms or abbreviations, unless they are commonly known and used like URL, HTTP, CPU, etc. SAGA_NumCpus should be SAGA_NumberCpus or SAGA_CPUCount. This makes it much easier to read in the source code then cryptic names. - The use of all upper case of lower case in naming is subject to discussion. But by habit all constants are in upper case, attributes and methods in lower case unless they are composite names. - Use of "_" in names is C style programming. In OO it is only used if upper or lower case mix naming cannot be used, for example a constant. So "byte_written" becomes "byteWritten" or "SAGA_JobCmd" would become "SAGA_JOB_COMMAND". Depending on who write each API you can see the writer\222s main coding language influence. For more information about code convention, SUN Microsystems has a good tutorial: http://java.sun.com/docs/codeconv/. Yes it is for Java but it can be applied to any OO based language or procedural language. - Typos: well there a number of typos to be removed. OK it is still a v0.2 ;) - In the stream API for the "write" and "read" methods. Why not add an 'offset' attribute to the calls? This might be language specific, but in Java for example you can not just shift the initial pointer like in C/C++ so the data always has to start at 0. Forcing to buffers to be used at index 0 all the time might not be welcome and additional programming overhead will be necessary to use the API. ====== GridLab: Application Migration ---------------------- +-----------------------------------------------------------------+ The SAGA API allows to migrate any job it can handle with the job class, using the migrate method. That provides an easy solution for the GridLab migration use case if supported by the implementation/middleware/backend: -------------------------------------------------------------- #include <saga.hpp> #include <vector> #include <string> using namespace std; int main () { saga::job_server js; saga::job j = js.run_job ("remote.host.net", "my_app"); job_definition jd = j.get_job_definition (); vector <string> hosts; vector <string> files; hosts.push_back (string ("near.host.net")); files.push_back (string ("http://remote.host.net/file > http://near.host.net/file")); jd.set_vector_attribute ("SAGA_HostList", hosts); jd.set_vector_attribute ("SAGA_FileTransfer", files); j.migrate (jd); cout << "Heureka!" << endl; return (0); } -------------------------------------------------------------- (Question: does the SAGA migrate call move checkpoint files automatically, or do they need to be specified in the new job description as above?) However, for the complete use case to be implemented on application level, a number of steps cannot be implemented in SAGA. The call sequence would be: In the application instance which performs the migration on the other job: - trigger migration for the remote job - discover new resource + move checkpoint data to new resource + schedule application on new resource + continue computation (and discontinue old job) In the application instance which gets migrated - get triggered from checkpointing = perform application level checkpointing - report checkpoint file location(s) Items marked with + possible to implement in SAGA - impossible to implement in SAGA = (currently) not related to SAGA. For the complete implementation of the use case, SAGA misses: 1) means to communicate with the remote application instance 2) means to discover new resources Notes: 1) means of communcation are actually given, but not per se usable for this use case. E.g. streams are a definite overkill for signalling checkpointing requests. Signals (as in job.signal (int signal)) would work, but only if the remote job uses signal handling as a checkpoint trigger. That also might be difficultato use if the job is running in a wrapper script, or in a virtual machine etc - that might not be transparent to SAGA, and would require direct communications. Also, the signalling method misses feedback about success of the operation, and cannot return information such as the location of checkpoint files. 2) the current SAGA API covers job submission to specific hosts, or lets the middleware choose a suitable host for submission. However, the brokering result is not exposed on API level, as would be neccessary for this specific use case, and possibly for other dynamically active Grid applications. One way to implement that is to provide a direct interface to Grid information systems, and on that way expose information about available resources. That would actually be more flexible, as is e.g. also allows the discovery of specific services, but would also require additional semantic knowledge on application lelvel. +-----------------------------------------------------------------+ ====== Univ of Vienna: Rainer Schmidt: --------------- I had a look at the short version of the SAGA spec. and send you some comments: I think the first tier covers all (and even more) of the general aspects of our use cases. Especially the file and job interfaces could be (partially) mapped well against our middleware. We require basic remote file handling, job submission, monitoring, and state inquiry. I'm not yet sure if I got Session and Context right but I think it would be feasible to integrate with our API and very helpful. I also like the idea of having an asynchronous API and the task interface. Of course, we would need to extend the API for accessing our VGE middleware specific services for e.g. service discovery, QoS negotiation or resource reservation. Hope this helps a litte! <<< use GridRPC? >>> ===== SCOOP [LSU] ------ really like the Session API and Logical Files API... would help a lot. Everything looks ok and should be extremely valuable to SCOOP. For SCOOP ver 2 our job submission interface was from Gram-job-manager -> Condor Master -> Condor Pool. Reason : Portal was limited to GRAM submissions only. One of the main difficulties we faced was interpreting of error codes. It was very confusing what the exact error code meant since there were layers of RMs. I wonder if SAGA can help resolve such complexities. :) <<< Comment Andre: -------------- Yes and no - if a SAGA implementation reports errors badly (maybe because it does not get good error messages from the middleware), we cannot do much apart from saying "Uh, bad!". However, at least you will have one and only one consistent and reliable way of error reporting on application level. The submission chain will be completely hidden. However, if your grid requires the chain, you still need to implement it _somewhere_, be it in a saga adaptor, or behind a gatekeeper... >>> Feedback from Anonymous: ---------- We will not being using the SAGA API for our use case. SAGA evolved into somethign very different from what we originally found potentially useful for our use case (three calls- authorize(), copy_file(), and run_job()). We aren't interested in POSIX-like file semantics, threads, or the complexity of the API. <<< Comment Andre: -------------- Hmm, I checked their use case. It is not trivial (e.g. includes data base access, steering and visualization, at least to some extent). Also, a 'three call auth' we have (actually, we have zero call minimum and 5 call maximum). run_job and copy_file we have as well, both need exactly 2 calls in total: { saga::directory dir; dir.copy (src, target); saga::job_server server (); server.run_job ("remote.host.net", "/bin/myjob"); } So, it might be that they percieve SAGA as too complex, and that should be kept in mind, but the criticism is at least not well formulated I think. >>> ======

1 0

Fwd (newby@arsc.edu): [wg-all] Ten documents now in public comment:
by Andre Merzky 04 Oct '05

04 Oct '05

Hi all, our UseCase Doument made into the public comment state, finally! Thanks to Shantenu to take care of that. PLEASE go to http://www.ggf.org/ggf_docs_public.htm and comment on it - w/o comments, the document will NOT advance in the process. Of course, the other documents listed below would also be happy about comments ;-) Cheers, Andre. ----- Forwarded message from Gregory Newby <newby(a)arsc.edu> ----- > From: Gregory Newby <newby(a)arsc.edu> > Date: Tue, 4 Oct 2005 06:57:43 -0800 > To: All WG/RG Members <wg-all(a)ggf.org> > Subject: [wg-all] Ten documents now in public comment: > > Ten documents have entered public comment. > > Please visit this link to read the documents, view public > comments, and make public comments. Public comments are a > very important part of the standards document process, and > your comments are valuable to authors in improving their > documents. > > Link: > http://www.ggf.org/ggf_docs_public.htm > > These documents are Recommendation track, and are in 60-day public > comment: > > 1. CDDLM Component Model > 2. Resource Usage Service (RUS) based on WS-I Basic Profile 1.0 > (the link for this document isn't added yet) > > This document is Community Practice, and is in 60-day public > comment: > > 3. Standardised Namespaces for XML infosets in GGF > > These documents are Informational, and are in 30-day public > comment: > > 4. OGSA Profile Definition 1.0 > 5. An architecture for GridCPR services and a GridCPR API. > 6. Use-cases to be addressed by GridCPR WG > 7. Grid Economy Use Cases > 8. Use Cases document for SAGA-RG > 9. Grid Scheduling Use Cases > 10. OGSI Authorization Requirements > > > Thanks to all the authors, editors, WG/RG chairs and members > for a highly productive document series! > -- Greg Newby, GGF Editor > > Dr. Gregory Newby, Chief Scientist (Acting), Arctic Region Supercomputing Ctr > Univ of Alaska Fairbanks-909 Koyukuk Dr-PO Box 756020-Fairbanks-AK 99775-6020 > e: newby AT arsc.edu v: 907-450-8663 f: 907-450-8601 w: www.arsc.edu/~newby > > ----- End forwarded message ----- -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

None
by Andre Merzky 04 Oct '05

04 Oct '05

1 0

Use Case Evaluation
by Andre Merzky 04 Oct '05

04 Oct '05

Dear group, meanwhile we got feedback from six use case authors, which evaluated the current SAGA API draft against their use cases. Some of them have been discussed in yesterdays GGF session. Comments from the list are of course welcome. Best regards, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

task model question
by Andre Merzky 04 Oct '05

04 Oct '05

Hi all, I added another issue, and would welcome comments. Cheers, Andre. 63) should there be a method: task.get_task_factory () which returns the tf which created the task? GridRPC has a similar method call, and it seems to make it much easier to keep track of tasks. E.g. one could start a remote read task, and when that is finished start another read: saga::file f (url); saga::file::task_factory ftf = file.get_task_factory (); saga::task t = ftf.read (100, out); t.run (); ... // do may things if ( saga::task::Done = t.get_status () ) { // ok, do another read on that one... saga::file::task_factory ftf = t.get_task_factory (); ftf.read (100, out2); } That would be VERY useful for task containers, e.g. if 1 of n tasks finishes, and requires a follow up call on the same task factory. Right now, the user needs to maintain a mapping from task to the respective task factory (which is not tooo difficult though, just possibly cumbersome). However: the problem with that approach is, that the return type for the call differs for the various task types!! So it would either require: - have specific task types, and have task container etc. work on a base class of them (ugh?) - have a base task_factory type which gets returned and casted (ugh?) -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

Minutes from SAGA-RG meeting (#1) at GGF15
by Thilo Kielmann 03 Oct '05

03 Oct '05

All, attached please find the notes from today's SAGA meeting. All errors and typos are solely mine. Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/

2 1

Ad-hoc BOF on Streams at GGF15 (fwd)
by Shantenu Jha 03 Oct '05

03 Oct '05

The repost didn't make it even a second time. I think... ---------- Forwarded message ---------- Subject: Ad-hoc BOF on Streams at GGF15 Date: Fri, 23 Sep 2005 14:16:29 +0100 From: "Dave Berry" <daveb(a)nesc.ac.uk> To: <saga-rg(a)ggf.org> Cc: "Allen Luniewski" <luniew(a)almaden.ibm.com> Dear SAGA RG, We are organising an ad-hoc BOF at GGF15 on the subject of data streams in Grid architectures. Our aim is to bring together people working on various aspects of data streams. Our personal goals are to determine how best to fit streams into the OGSA Data architecture but we also see this session as an opportunity for people to share experiences and meet others working in overlapping areas. We hope that some of you will come along and contribute to the discussion. In particular, we would welcome a speaker from the SAGA RG. Please contact me if you would like to speak. The BOF will be from 6.00-7.30pm on Monday Oct. 3rd. The first half of the session will see presenters addressing some of the questions below. We will keep the second half of the session for discussion. Issues to consider: - What do we mean by a "data stream"? - Grids have to consider attributes such as security, persistence, and quality of service. Do these raise any particular questions for streams? - What are the important use cases? Are there unusual or non-obvious cases that we need to consider? - What is the natural programming model for streams in a distributed environment? - How do we connect one stream to another? - How do streams fit into a service-oriented architecture? - The OGSA Data Architecture currently has sections on Provisioning, Querying, Replication, Caching, etc. Do these apply to streams as well as static data? Are there any particular aspects of streams that affect any of these sections? We look forward to seeing you there. Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0