May 2006 - saga-rg - lists.ogf.org

ISSUE 132
by Andre Merzky 22 Jun '06

22 Jun '06

132) when do we refer to target host name, and when to resource manager contact, on job submit/description? Se mail thread with Graeme/Chris "JobService.submitJob() query" - OPEN, URGENT See also the comment from Graeme to the list today below: > Andre, > > The client must be able to specify the resource manager to which jobs > are submitted, as a service URL or machine name. This should be a > argument for either the JobService constructor, or the > JobService.createJob() & JobService.runJob() methods. > > I guess that conceptually I prefer specifying the resource manager > during the construction of the JobService object. However in an earlier > mail Chris Smith indicated that the 'host' argument of the runJob() > method was originally intended to indicate the machine upon which the > job should be run (mapping to an element of the SAGA_HostList > attribute). This is confusing and the 'host' argument should be removed, > or redefined to indicate the resource manager. > > Graeme I propose to go with Grames proposal unless there are opinions to do differently. I like the version to redefine the host arg better than to remove it altogether. Other opinions? Thanks, Andre. -- "So much time, so little to do..." -- Garfield

2 4

Re: [saga-rg] ISSUE 90
by G.E.POUND＠soton.ac.uk 22 Jun '06

22 Jun '06

Andre, Please find attached comments on the Exceptions thrown by the methods of JobService and Job in the Java bindings (including changes that I must implement). Most are quite straight-forward, apart from JobService.createJob(). Graeme Quoting Andre Merzky <andre(a)merzky.net>: > That would be useful! Malcolm should also have some idea on > what error handling is exposed by his implementation, that > might be useful as well. > > Our implementation has no job so far *sigh*. > > Thanks, Andre. > > > Quoting [G.E.POUND(a)soton.ac.uk] (Apr 20 2006): > > > > I can feedback the exceptions that I found essential from the Job > package in > > the Java bindings. > > > > Graeme > > > > > > > > Quoting Andre Merzky <andre(a)merzky.net>: > > > > > 90) exceptions missing in job! > > > - OPEN, URGENT > > > > > > Well, that says it all. > > > > > > Volonteers? > > > > > > Cheers, Andre. > > > > > > > > > -- > > > "So much time, so little to do..." -- Garfield > > > > > > > > > > > > > > > > > > > > > > > > -- > "So much time, so little to do..." -- Garfield > > >

2 1

Requirement Document - Group feedback needed...
by Andre Merzky 22 Jun '06

22 Jun '06

Hi Group, the SAGA Requirement document is, finally, getting close to publication. Attached is the current version, which we intent to submit to GGF Office on Friday night. This is the 'last call' for the group to provide feedback on the draft, or (preferrably ;-) to give us a 'go ahead'. Two notable points might change before submission: the author list might get extended (please speak up if you feel left out, or have suggestions!); and the Java-CoG sections needs revision - we are waiting for input from Gregor on that. Please provide feedback as early as possible, to allow us to keep our Friday submission deadline. Comments after (local) businness hours are very likely to be ignored (for this version). Best regards, Shantenu and Andre. -- "So much time, so little to do..." -- Garfield

3 3

Minutes from SAGA-CORE-WG at GGF17, 1st session
by Thilo Kielmann 20 Jun '06

20 Jun '06

Meeting of SAGA-CORE-WG at GGF17 1st meeting, may 11, 3:45 - 5:15 Session chaired by Andre Merzky Notes taken by Shantenu Jha, Thilo Kielmann Agenda: - agenda bashing - work on issue list for the SAGA1.0 spec Andre opens the session, circulates the sign-up sheet, referring to GGF's IPR policy. agenda bashing: --------------- the agenda is accepted "as is" by the participants open issues from "the list": ---------------------------- job states ---------- the state diagram is revisited Q: remove (or defer) job.checkpoint ? (->gridcpr) what about job.migrate ? the discussion is leaning towards a 2 layer state model: - a generic level - detailed BES model the differences revolve around "hold" states of jobs. Queueing systems do provide them, but applications seem to have little (no?) use of it. (see: SAGA use cases) How much should a SIMPLE API (-> "S"AGA) expose? It is clear that SAGA states should be compatible with BES, however, possibly only have a proper subset in the generic level, and the full detail in the detailed level. action item : list the options, come to rough consensus have layered model Andre to place discussion both (bes and saga) mailing lists further issues from the issue list that were discussed: 12, yes change 29, see 2nd session (Stephan Hirmer's talk) 30 ACLs Q: are all ACL systems equal? e.g., NFS ACL vs WinFS vs Solaris AFS and will that cause problems? Andre agrees: SAGA has an omission - no provision for distinguished names Line 81: strawman_namespaces ACL_Admin ==> replace with ACL_Owner?? no! "admin" describes better the role of maintaining; these could be multiple users, not just the single owner Use Base Case approach for security? e.g. look at all the diff OS there is commonality and then specific issues Andre: to post on the mailing list.. 36, language examples missing (not discussed) 47, jsdl compliance and 88, clean up files JSDL has the notion of which files to get rid of after execution, but support for this feature will be limited. JSDL: keep, remove, don't care (default) Andre: Add an attribute "clean_up" 55, references, not discussed editing "only" comments skipped 102, what classes need error handler ErrorHandle -- defined for languages like C - that do not have an error interface. interface error_handle provides the ability to rethrow errors ! provides guidelines for how procedural languages should handle errors. check "has error type" ??? -> ask Tom -> add errorhandler interface to task get_Error has_error 108 constness of operations do it in prosa, in/out/inout remain as in SIDL 111 file constructors do open implicitly agreed. 132, bind resource manager to jobservice (in constructor) runjob needs to be discussed on the mailing list Andre promises to buy beer for people willing to help writing/editing the spec doc. (From the note taker: it was not clear whether the beer should be paid before or after editiing the document.) At 5:15, the session is closed.

2 1

sessions at GGF17...
by Andre Merzky 13 Jun '06

13 Jun '06

Hi Folx, the sessions scheduled for GGF17 are: Wednesday, May 10 1:45 pm - 3:15 pm, room G-408 Wednesday, May 10 3:45 pm - 5:15 pm, room G-408 Thursday, May 11 3:45 pm - 5:15 pm, room G-408 Friday, May 12 11:00 am - 12:30 pm, room G-408 The agendas we originally proposed are somewhat out of sync with the status of the group work, so we would like to propose the following agendas. Feedback is appreciated. We intent to publish the agenda on the GGF web pages on Sunday morning. It is recommended to read both the SAGA Requirement Document and the SAGA API specification prior to the sessions. That is not to scare people away if they don't read the docs, but efficiency of the meetings will certainly improve with familarity with the work done in the group so far. The agendas below also imply that reading of other GGF specs and drafts (JSDL, BES, DRMAA) would be very beneficial. Cheers, Andre. Session 1 (RG): --------------- The new SAGA-RG incarnation will discuss administrativa, and plan collaboration with other GGF groups (RPC, CPR, SAGA-CORE, ...). A short overview presentation of the current SAGA API is included for bypassers. Agenda: - administrativa - SAGA overview for the crowd (3 slides), announce other sessions etc - evaluate the SAGA REQ document, and derive RG scope out of that - sync with RPC and CPR, finalize on design teams Session 2 (RG): --------------- The SAGA-RG will flesh out the work of the RPC and CPR design teams, and work on synchronization with other GGF groups in general. Agenda: - plan for real req. document for these design teams - work on SAGA-RPC draft - work on SAGA-CPR draft (if available) - synchronize with (and plan for future) new specs in GGF (GFS, BES, JSDL, ...) (we should READ them before GGF) - considerations about a SAGA look-&-feel spec (based on the look-&-feel parts of the core spec) Session 3 (WG): --------------- - discuss parts of API which should be moved into the RG look-&-feel document (object, task, ...) - update on issue list, try to close remaining items - define timeline/responsibilities for finalization (editing, proof reading, consistency checks) - define timeline/editors for cook-book Session 4 (WG): --------------- - continue on open issues if needed - define editors and format for language bindings - update on implementations - discuss bulk optimizations -- "So much time, so little to do..." -- Garfield

7 14

Minutes from SAGA-CORE-WG at GGF17, 2nd session
by Thilo Kielmann 31 May '06

31 May '06

Meeting of SAGA-CORE-WG at GGF17 2nd meeting, may 12, 11:00 - 12:30 Session chaired by Andre Merzky Notes taken by Shantenu Jha, Andre Merzky, Thilo Kielmann Agenda: - agenda bashing - update on implementations - bulk operations (Stephan Hirmer) - issues from the issue list - document editing business - GridCPR update on implementations ------------------------- - DEISA, nothing new (files+jobs working and stable) - OMII, (Java) great progress, good feedback, following spec changes available on their website Java bindings would need external cross-check from a Java expert Ed checks if this can be done at LSU - SAGA-A (C++), added RPC subsystem, added adoptors, almost complete/done for Windows and Linux, GT4 adaptors SAGA-A is available from the saga cvs via anonymous cvs checkout Boost licence (GPL) Windows and Linux Both Windows and Linux versions are available LSU/CCT cvs bulk operations (talk by Stephan Hirmer) ---------------------------------------- Stephan reports on his work at LSU, implementing bulk operations within SAGA-A. searching for "similar" tasks inside a task container that can be run as a single bulk operation first tests show promising speedups -> fesibility of bulk ops issues: should task_container be derived from task (grouping of tasks)? should templates be allowed in API classes? Implementor's note - Templates would be useful issues from the issue list -------------------------- 136 - Errors & task_container motivation seems to come from workflows no use-cases for workflow, hence leave at status quo 137 tasks survive app life time? problem: do (web) service invocation that outlives app. runtime (like long running file transfers) tricky between tasks (threads/asynchrony) and jobs (remote user programs) defer beyond saga 1.0, not yet well understood to implement problem: what to do with pending tasks at app. end ?? -> undefined in standard w.r.t. implementation document editing business ------------------------- people willing to proofread parts of the SAGA spec: Derek Simmel reviews files and jobs Yusuke reviews task model Pascal streaming, logical files until end of May cook book: need examples, implementors, into the Wiki language bindings: c++ as headers java as interfaces + javadoc GridCPR ------- a separate design team will propose a subsystem based on Derek's proposal At 12:35, the session is closed.

1 0

Is there Factory in Java Binding?
by Hidemoto Nakada 24 May '06

24 May '06

All, I have a question on the Java Binding. Does the current Java binding include explicit factory notion? Factory mechanism is important to - reuse a source code with several implementation of the API without modification of the code. - leverage several implementation simultaneously from one client code. One good example of factory design can be found in the Dom parser definition; org.w3c.dom.DomImplementation. When you create a Dom document - get a DomImplementation from org.w3c.dom.bootstrap.DomImplementationRegistory - then create a Document object using DomImplementation. This mechanism might be too complicated for SAGA, but I think some flavor of this is required. -hidemoto

4 5

SAGA Documents published
by Andre Merzky 17 May '06

17 May '06

Hi folx, we proudly announce that we got 2 documents advanced in the GGF document pipeline: - the "SAGA Use Case Document" is now officially publised as GFD.70 (see http://www.ggf.org/gf/docs/?final) - the "SAGA Requierements Document" is now in 30 day public comment period (http://www.ggf.org/gf/docs/?public_comment) Thanks to the authors, and to the folx who gave feedback and input. In particular, thanks again to the use case authors!! For the requirement doc, we would very much appreciate if you would take a couple of minutes and actually comment on it. Preferably you have the time to read it, and criticise it of course. But also if you are already familiar with it, or only have time/interest for parts of it: please comment, and be it only 'agree with publication' or 'is crappy'. https://forge.gridforum.org/forum/forum.php?forum_id=613 Without any comments, the doc will NOT advance (and obviously also not improve)! So please spread the word. Thanks, Andre. PS.: please excuse the cross posting... -- "So much time, so little to do..." -- Garfield

1 0

Fwd (theimer@microsoft.com): RE: [ogsa-wg] Thoughts on extensions mechanisms for the HPC profile work
by Andre Merzky 17 May '06

17 May '06

Hi SAGA, see below a mail quited from Marvin Theimer, which went to the OGSA list. I think his points about implementation of optional features are very valid ones, and I would like to add a similar statement to the spec, like: "A saga object or method MUST be implemented with exactly the semantics specified, or not at all." Any thoughts on that? Cheers, Andre. ----- Forwarded message from Marvin Theimer <theimer(a)microsoft.com> ----- > Subject: RE: [ogsa-wg] Thoughts on extensions mechanisms for the HPC profile work > Date: Thu, 11 May 2006 18:16:01 -0700 > From: Marvin Theimer <theimer(a)microsoft.com> > To: Steve Loughran <steve_loughran(a)hpl.hp.com>, ogsa-wg(a)ggf.org > > Hi; > > I agree with you that we should try to keep to as simple a means of > specifying the concept of "must understand" or "may ignore" as possible. > I haven't looked at the difference between how soap1.1 and soap1.2 do > things and will go off and do my homework. > > One thing that I believe is happening in this conversation is that two > separable issues are being convolved together. One issue is whether or > not to allow optional feature specifications. The other has to do with > the main concern that I heard in your email, namely that "optional" > features are often ill-defined in terms of their semantics. I would > argue that the semantics of optional features need to be precisely > defined -- including what it means if the feature is optional. I would > argue that ANY feature that isn't precisely defined in a spec has the > potential to cause interop problems. So I would argue that "must > understand" features must be implemented precisely and completely. > Alternatively, "may ignore" features must be either completely and > precisely implemented or not at all (meaning the semantics provided must > be those if the feature had not been mentioned). > > Marvin. > > > -----Original Message----- > From: owner-ogsa-wg(a)ggf.org [mailto:owner-ogsa-wg@ggf.org] On Behalf Of > Steve Loughran > Sent: Tuesday, May 09, 2006 8:12 AM > To: ogsa-wg(a)ggf.org > Subject: Re: [ogsa-wg] Thoughts on extensions mechanisms for the HPC > profile work > > Marvin Theimer wrote: > > Hi; > > > > The beauty of a "may ignore" flag is that it can be ignored. :-) So, > > whereas I side with Donal in believing that complex optional resource > > descriptions rarely work well, I side with Karl in believing that the > > flag should be available at the protocol level (as an extension case > > since I'm trying to be hard-nosed about not letting anything creep > into > > the base case that isn't absolutely necessary for minimal interop). > > > > It's worth noting that the concept of "optionally understood" > parameters > > is something that many consider to be one of the bigger success > stories > > of the way the Web works (as compared to Web services). > > > > > Marvin. > > > > > Sometimes. But then there is the way that you have to comment out stuff > in script tags in case a legacy browser renders your javascript, which > strikes me as a failure mode of the HTML extension system: > > <script></script> > > > In SOAP1.1 the mustUnderstand logic is simple and pretty much all you > need. SOAP1.2's MU logic is way more convoluted, and that makes it a dog > > to test -which, in a TDD process, means a dog to implement, and will > inevitably a source of problems for years to come. > > One issue with mustUnderstand logic, is what does "understand" mean? > Does it mean "recognise", or does it mean "process in 100% compliance > with the official specification of what this soap header required". > Axis1.1 shipped with the check for all headers being understood taking > place *after* the message was delivered. While following the letter of > SOAP1.1, and passing the limited (stateless) tests of SOAPBuilders, it > violated the spirit of the spec quite blatantly. > > It's testing those optional bits that really hurts. Unless the clients > implement all possible bits of optional behaviour, you cannot tests that > > the endpoints implement it properly. This essentially makes it > impossible to make any declaration about the quality of implementation > of any optional part of a spec -you have to just hope for the best. > > summary: whenever you mark something as optional, you create an interop > problem. > > -Steve ----- End forwarded message ----- -- "So much time, so little to do..." -- Garfield

2 1

Minutes from 2. session of SAGA-RG at GGF17
by Thilo Kielmann 17 May '06

17 May '06

Meeting of SAGA-RG at GGF17 2nd meeting, may 10, 3:45 - 5:15 Session chaired by Andre Merzky Notes taken by Andre Merzky, Thilo Kielmann Agenda: - agenda bashing - questions (misc.) to the SAGA group - GridRPC interface - SAGA look-and-feel document Andre opens the session, circulates the sign-up sheet, referring to GGF's IPR policy. agenda bashing: --------------- the agenda is accepted "as is" by the participants questions (misc.) to the SAGA group: ------------------------------------ Q: what do i need for saga? globus + saga package a,b,c ?? -> check acs+cddlm groups -> saga cookbook Q: relation with other ggf groups? -> see requirements doc ----------------------------------------------------------------------------- GridRPC interface: ------------------ - open problems with rpc in SAGA: - config file imple specific - still open, follow up on list - error returned either on init or on call - same in other places in SAGA, so no problem - needs clean up in SAGA eventually? - Ninf-G allows to bind handle to multiple methods ('module') - delay for later iterations, of no concern right now - variable parameter list - no issue, define list of char^, and allow language binding to do vararg some detailed comments in the code below ---------------------------------------- The SAGA rpc proposal: ---------------------- namespace saga { ///////////////////////////////////////////////////////// // // rpc // class rpc { public: rpc (session const & s, const std::string & name); rpc (const std::string & name); ~rpc (void); // call void call (std::list <char *>); }; // rpc } ---------------------------------------------------------- // most trivial example, controversity about config file, as that // is implementation specific. Two options: // - if the config file is not needed to be changed by the end // user, hide its existence in the implementation // - if the config file needs to be changed by the end user // (during runtime), provide it to the constructor as optional // URL parameter - that URL could be obtained by a resource // discovery later. #include <saga.hpp> int main () { try { saga::rpc handle ("dgesv", uri); // see comment above } catch ( const saga::exception & e ) { std::cerr << "Error creating function handle: " << e.what () << std::endl; } } ---------------------------------------------------------- ---------------------------------------------------------- // This bases on a sample program for Ninf-G, calculating pi // it shows the syncroneous version of call. #include <saga.h> std::string func_name = "pi/pi_trial"; std::string config_file = "client.conf"; int port = 0; int main (int argc, char *argv[]) { if ( argc != 2 ) { std::cerr << "\n\tUSAGE: " << argv[0] << " <times>\n\n"; exit (2); } long times = atol (argv[1]); try { // Initialize Function handle saga::rpc handle (func_name); // Synchronous call, here using varargs long answer; handle.call (0, times, &answer); // Compute and display pi std::cout << "PI = " << 4.0 * ((double) answer / times)) << std::endl; } catch ( const saga::exception & e ) { std::cerr << "Catched saga error: " << e.what () << std::endl; exit (2); } return (0); } ---------------------------------------------------------- ---------------------------------------------------------- // similar example, but use async calls #include <saga.hpp> string func_name = "pi/pi_trial"; int main (int argc, char *argv[]) { if ( argc != 2 ) { std::cout << "\n\t"; std::cout << "USAGE: " << argv[0] << " TIMES\n"; exit (2); } // initialize vars from CL arguments long times = atol (argv[1]); std::vector <std::string> hosts; try { // create result vector and task container std::vector <long> count; saga::task_container tc; for ( int i = 0; i < times; i++ ) { // create rpc handle saga::rpc handle (func_name); // perform async call, returns task in Runnning state saga::task t = handle.call <saga::async> (i, &count[i]); // add task to task container tc.add (t); } // wait tasks (for ever, for all) std::list <saga::task> finished = tc.wait (-1.0, saga::task::All); // the same would be (default parameters): // std::list <saga::task> finished = tc.wait (); // Compute and display pi. As all tasks are done (error test // missing!), we can use the output vars int sum = 0; for ( i = 0; i < times; i++ ) { sum += count[i]; } std::cout << "PI = " << 4.0 * (sum / ((double) times))) << std::endl; } catch ( const saga::exception & e ) { std::cerr << "Found saga error: " << e.what () << std::endl; exit (2); } return (0); } comments to this example: - Q: what is the life time of task? A: defined by runtime of task, or limited by cancel - Q: does cancel guarantee resource de-allocation? A: No, right now it does not block, and implementation is doing best effort, but no guarantees - after some discussion we agreed to add a timeout to cancel(), with -1 meaning forever, to allow to wait for guaranteed resource deallocation on task cancelation. Default will be no timout (0.0). ---------------------------------------------------------- - TODO items: - clean up strawman - post on lists - re-iterate on the config file problem - implement - done :-) ------------------------------------------------------------------------------ SAGA look-and-feel document: ---------------------------- scope of a look&feel document: - object - tasks - error/ tasks - session/context - attributes everything non-functional - coherent naming At 5:10, the session is closed.

1 0