February 2006 - saga-rg - lists.ogf.org

JobService.submitJob() query
by Graeme Pound 07 Mar '06

07 Mar '06

Hi, Can anybody clear up this issue for me? I may be missing something in the spec. Thanks, Graeme -3.31 How is the resource manager endpoint for JobService.submitJob() specified? The endpoint is specified by an argument of JobService.runJob(), but is not _obvious_ for submitJob(). Is the endpoint a property of the instance of JobService (with runJob() a 'static' method of the class), or is the endpoint specified as an attribute of JobDefinition? I assume that the contents of the JobDefinition attribute 'SAGA_HostList' is intended to be passed to the resource manager (rather than specify the resource manager itself); for example mapping to the 'CandidateHosts' element of a JSDL document. Is an additional attribute within JobDefinition required to specify the endpoint?

4 17

1.4 version of the spec
by Andre Merzky 07 Mar '06

07 Mar '06

Hi all, the lates version of the strawman (1.4) is on the wiki. Cheers, Andre. -- "So much time, so little to do..." -- Garfield

2 5

Clean Up after jobs - issue 61
by Andre Merzky 06 Mar '06

06 Mar '06

Outside the SAGA use cases, we got feedback from some medical people about the job API. Open questions in the discussion have been: - When are files created by a job cleaned up? - Can we call clean_up? - Can specify files to clean up, if default is to leave them be? Their concern is mostly privacy: they want to ensure that no data a job worked on are left behind. Does anybody have insight into the semantics of the BES clean-up stage, or of other job managers? Do other people also see the need for explicit clean-up? If not (e.g. no answer in one week), I would leave that issue for later consideration, as we get more specific use cases in that respect. Cheers, Andre. -- "So much time, so little to do..." -- Garfield

2 1

CVS info
by Andre Merzky 27 Feb '06

27 Feb '06

Well, the CVS to Mail thginkg does not seem to work, yet, so this is an excerpt for what happened in CVS (on the strawman) in the last two weeks (time reversed): ---------------------------- date: 2006/02/17 15:03:20 worked on the issues resolved in Athens ---------------------------- date: 2006/02/15 11:15:42 commit changes resulting from yesterdays saga session ---------------------------- date: 2006/02/11 04:00:18 issue 59 done (attribs to stream now well defined ---------------------------- date: 2006/02/08 12:54:39 same game: worked on 2 issues, added 3 new one, w/o finishing the old ones... *sigh* - done: 27, 113, 116 - new: 117, 118, 119 Started to clean up monitorables, and added stream example. Constructors added to all SIDLs, session as opt. arg missing, some detailed descr. missing (new items) ---------------------------- date: 2006/02/07 11:32:33 fixed some pdf formatting (i62) remove void signatures (i85) started to work on the constructors/destructor (i65) added some new issues Summary: 2.5 issues done, 5 issues added. No, no convergency in sight... ---------------------------- date: 2006/02/06 13:46:13 added an iisue about unknown attributes ---------------------------- date: 2006/02/04 15:18:50 splitted the issue list into OPEN and CLOSED issues, to simplify search. Cheers, Andre. -- "So much time, so little to do..." -- Garfield

1 1

Re: [saga-rg] 1.4 version of the spec [part2]
by Graeme Pound 24 Feb '06

24 Feb '06

Andre, Your reply did not go to the mailing list. Graeme Andre Merzky wrote: > Quoting [Graeme Pound] (Feb 24 2006): >> Andre, >> >> Further comments on revision 0.4 (not all negative ;-) > > Phew! :-) > > >> Thanks >> Graeme >> >> >> #1 submitJob() has been renamed create_job(), however there is still >> no endpoint defined for this method - this is broken > > Right, fixed in CVS now. > > >> #2 submitJob()/create_job(). The described behaviour of this method >> has not changed; is this merely a superficial change, or is it intended >> to facilitate the separation of Job creation and submission at a later date? > > It is! job.run (); triggers the submission. > > The job part is missing a number of description and semantic > notes. Also, it is missing exceptions completely - its on > the TODO list *sigh* > OK, I missed the Job.run() method. > >> #3 Regarding the position upon getter/setter methods. >> >> "Many SAGA objects have very frequently use attributes. To >> simplify usage of these objects, setter and getter methods MAY >> be defined by the various language bindings, additionally to >> the interface described below" >> >> I appreciate this statement. I have been pondering this issue and >> believe that getter and setter methods are appropriate for the Java >> bindings. In addition to the issue of typesafety they allow the >> semantics defined by these attributes to be versioned much more >> explicitly. When the SAGA API changes (infrequently ;-) a client using >> an unsupported attribute via the getter/setter will receive a compile >> time error, otherwise the attribute will be hardcoded as a string and >> produce a runtime error. This is less of an issue for dynamic languages >> such as Python. > > Agree. > > >> #4 Should the close() method be defined as a method of ns_entry? As I >> mentioned in an earlier email it is not applicable to Directorys (which >> implement ns_entry), and perhaps not to logical_files or >> logical_directory. I believe that close() should be defined for classes >> as required. > > As said before, I am not sure about this. Close was > requested as means of freeing the resource explicitely (i.e. > timely) on systems with garbage collection (Java ;-). Maybe > 'destroy' would have been more appropriate, but that again > would clash on real files and streams, as 'close' is 'the > right thing' on those. > > I don't have a better idea right now. Out? In? *shrug* > > >> -3.33 The uniqueness of Context within a Session is a little >> ambiguous. Defined as "a single Context instance can get attached only >> once to a specific Session instance". Whilst it would be possible to >> determine whether the same instance of a Context object has been added >> to a session, two different Context instances could describe the same >> security information. What is the intention of this restriction? Is >> there a better way to specify the uniqueness of Context objects; perhaps >> based upon contextType? (see 3.34) > > I pondered about that as well some days ago. I don't see a > good reason for that statement anymore. It simply should > not matter if a context gets added twice. > > The only potential problem is confusion with remove_context, > as that should certainly remove only one instance - so a > double context would be still attached on the session after > a single remove call. I don't think that this is a problem > though. > > I would prefer to delete the statement, if there ar no > objections... > > >> -3.34 Session.removeContext(Context context) is a little awkward. The >> client would have to retrieve the instance Context in order to specify >> that it should be removed (see 3.33). Could Context instances within a >> Session have labels, removal could then be based upon label? > > We do have cookies for callbacks, so we could also have them > for contexts, sure. But then also for tasks (in > task_container)? > > I guess its a matter of taste - I can imagine use cases for > both versions. But one version is easily mapped to the > other on in application space. > > >> ######### Comments on the Strawman API document >> >> -4.18 Not all methods are documented in revision 0.4, including; >> File.read_p() >> File.write_p() >> File.read_e() >> File.write_e() >> File.modes_e() > > Right, thats on the issue list. > > >> -4.19 Buffer arguments should be described as byte arrays, not strings >> in File and Ivec. > > Oh, did I miss that? I though I fixed all the buffer > strings... - will check. > > Hmm: > > struct ivec > { > long offeset; // position of data to be r/w > long length; // number of bytes to be r/w > array<byte,1> buffer; // data to be r/w > long result; // number of bytes r/w > } > > Hmm, is byte array in CVS. I am wondering if we look at the > version? > Sorry that is an anachronistic comment from revision 0.3 that slipped in. > >> -4.21 Typo in ivec "offeset" should read "offset" > > Aehm, thanks :-) > > >> -4.22 removeContext "Security Context to add" should read "Security >> Context to remove" > > Fixed. > > >> -4.23 removeContext "DoesNotExists" should read "DoesNotExist" > > fixed. > > Thanks, Andre. > > >> Andre Merzky wrote: >>> Hi all, >>> >>> the lates version of the strawman (1.4) is on the wiki. >>> >>> Cheers, Andre. >>> >>> > > >

2 1

Fwd (llorente@dacya.ucm.es): [drmaa-wg] New release of the GridWay meta-scheduler
by Andre Merzky 24 Feb '06

24 Feb '06

FYI, another DRMAA implementation arrived. Cheers, Andre. ----- Forwarded message from Ignacio Martín Llorente <llorente(a)dacya.ucm.es> ----- > Date: Fri, 24 Feb 2006 09:40:40 +0100 > From: Ignacio Martín Llorente <llorente(a)dacya.ucm.es> > Subject: [drmaa-wg] New release of the GridWay meta-scheduler > To: drmaa-wg(a)gridforum.org > > Dear friends, > > The new development release of the GridWay meta-scheduler is available > for download under the terms of the Apache License, Version 2.0 (see the > full announcement below). > > GridWay now provides full support for DRMAA to develop distributed > applications (C and JAVA bindings) on a grid consisting of distinct > computing platforms (SGE, PBS, LSF?) managed by Globus services. DRMAA C > implementation passes the DRMAA compliance testsuite and DRMAA JAVA > implementation has been tested with the howtos provided by Sun Microsystems. > > Please visit the DRMAA References and the Application Developer Guide at > http://www.gridway.org/guides.php > > Do not hesitate to contact me if you need any additional information. > > Sincerely, > > Ignacio M. Llorente > > --------------------------------------------------------------------------------------- > NEW DEVELOPMENT RELEASE (4.7.0) OF THE GRIDWAY META-SCHEDULER > > The GridWay team is pleased to announce that a new development release > (4.7.0) of the GridWay meta-scheduler is available for download under > the terms of the Apache License, Version 2.0. > > ABOUT THE GRIDWAY META-SCHEDULER: > > GridWay, on top of Globus services, enables large-scale, secure, > reliable and efficient sharing of computing resources (clusters, > computing farms, servers, supercomputers...), managed by different DRM > (distributed resource management) systems, such as PBS, SGE, LSF, > Condor..., within a single organization (enterprise grid) or scattered > across several administrative domains (partner or supply-chain grid). > GridWay is an open-source meta-scheduling technology that provides end > users and application developers with a scheduling framework similar to > that found on local DRM systems, allowing to submit, monitor, > synchronize and control jobs by means of a DRM-like CLI (gwsubmit, > gwwait, gwcancel, gwkill...) and the DRMAA API (GGF standard). > > HIGHLIGHTS: > > - Easy deployment and adaptability > - Improved efficiency and robustness > - Advanced scheduling capabilities, such as dynamic discovery & > selection, opportunistic migration, performance slowdown detection and > support for self-adaptive applications. > - Information drivers to interface MDS2 (MDS schema), MDS2 (Glue schema) > and MDS4 > - Execution drivers to interface pre-WS GRAM and WS GRAM (even both > simultaneously) > - Transfer drivers to interface GridFTP and RFT. > - Full support for DRMAA (GGF standard) to develop distributed > applications (C and JAVA bindings). > - Extended DRM-like commands to submit, monitor, synchronize and control > jobs; and monitor grid resources > - Extended documentation and improved support forum > > RELEVANT LINKS FOR THIS RELEASE INCLUDE: > > - Functionality: http://www.gridway.org/functionality.php > - Benefits: http://www.gridway.org/benefits.php > - Release Notes and Download: http://www.gridway.org/download470.php > - Documentation: http://www.gridway.org/guides.php -- "So much time, so little to do..." -- Garfield

1 0

Final Call Issue 15: The URL Problem
by Andre Merzky 23 Feb '06

23 Feb '06

Hi group(s), We discussed issue 15 in Athens, but could not come to consensus. I further discussed the issue with Thilo, Wolfgang Ziegler, and some Unicore guys (Philip and Achim). In order to settle the issue for the SAGA-CORE-API (NOT forever), I want to propose the following solution. If no reply comes within a week, I assume its accepted :-) - A URL utility class seems useful, but would increase the syntactic load on the current set of methods (either allow all methods with strings _and_ urls, or require transformation from strings to urls _always_ - both seems suboptimal). - hence NO URL utility class is mandated. - a NSEntry utility call is introduced: convert (in string target, out string new_url); Usage examples: saga::file f ("ftp://ftp.net.org/pub/data/file.txt"); std::string n1 = f.convert ("ftp://ftp.net.org:123/"); std::string n2 = f.convert ("ftp://alias.net.org/data/"); std::string n3 = f.convert ("http://www.net.org"); - that call tries to translate URLs, and if unsuccessfull returns an empty string. I know that is 'somewhat' of a compromise. A full fledged URL class certainly would be useful, and Java for example already has it. I would expect that most SAGA implementations will provide one anyway. But as we could not reach consensus on that, the topic seems to need more discussion, and more time which I wthink we don't have for stabilizing the API. The convert method could be implemented as follows (all services hosted on same resource): Known: 1 host: ftp.net.org proto: ftp root: /var/www/htdocs/ftp_store/ 2 host: ftp.net.org proto: ftp:123 root: /var/ftp_test/ 3 host: www.net.org proto: http root: /var/www/htdocs/ input: url: ftp://ftp.net.org/pub/data/file.txt tgt: http with 1 translates into absolute path abs: /var/www/htdocs/ftp_store/pub/data/file.txt with 3 translates into out: http://www.net.org/ftp_store/pub/data/file.txt Cheers, Andre. -- "So much time, so little to do..." -- Garfield

1 1

Queries and minor points
by Graeme Pound 23 Feb '06

23 Feb '06

Please find below a couple of queries, and diverse minor points, which arose during my preparation of a Java implementation of the API. I anticipate that my comments on the semantics of the API will become a little more substantive in due course. Thanks, Graeme ######### Queries - I am little unclear of the purpose and operation of Attribute.getVectorAttribute() and Attribute.setVectorAttribute(). What is returned if a vector attribute is queried with getAttribute()? Why are a vector of values associated with a key required? - contextType does not appear to be very extensible, how would different security models be specified? ######### Minor points - Exceptions are not described in the SIDL interface, however Exception types are listed in long description of the methods (for example ReadOnlyAttribute and NoSuchKey). This information should be in the SIDL description as it is fundamental to the interface. - capitalise ivec, openDirFlags, openFlags? - JobService.runJob() returns Job and stdin, stdout, stderr. However stdin, stdout, stderr are available from Job.getJobInfo()? Can runJob() simply return an object Job? - enum values are not specified by JobState - LogicalFile argument types not specified for addLocation, removeLocation, replicate. These should be 'string' - The enumerations openDirFlags and openFlags exist in the following packages SAGA.Namespace, SAGA.File and SAGA.LogicalFile (although SAGA.Namespace is imported by SAGA.File and SAGA.LogicalFile). Is this duplication required? - the interfaces Stream and StreamServer implements Attributes, however Attributes does not exist. Is this Attribute (perhaps Attributes is a better name for this class)? - File.readP() and File.writeP() define input argument of type 'pattern'. This type is not defined by the SIDL API, is 'pattern' an undefined class or primitive type? - in Stream void getContext (out context info); should read void getContext (out Context info); - strawman_error.txt: "All objects in SAGA implement ErrorHandler...", the SIDL for these objects should indicate that they implement the ErrorHandler interface. The following objects: Directory File Job JobService LogicalDirectory LogicalFile Multiplexer Stream StreamServer Task TaskContainer ... any others ?

4 5

Fwd (replogle@ggf.org): [gf-chairs] GGF17 Group Session Scheduling is Open!
by Andre Merzky 23 Feb '06

23 Feb '06

Hi Group(s), GGF was yesterday, kind of, but here we go planning for the next one already... I would very much like to see our 3 docs submitted as drafts by the March 31st deadline (use cases revised, req. initial, saga-core-api initial). Its difficult to plan for the sessions - anyway, I think the 3 session scheme we had over the last 2 GGFs worked pretty well. So I would propose to continue with that schema. However, we should also keep in mind that our ADs plan a application workshop, which surely will benefit from SAGA participation :-) Cheers, Andre. ----- Forwarded message from Joel Replogle <replogle(a)ggf.org> ----- > To: gf-chairs(a)ggf.org > From: Joel Replogle <replogle(a)ggf.org> > Subject: [gf-chairs] GGF17 Group Session Scheduling is Open! > Date: Thu, 23 Feb 2006 13:11:53 -0500 > > GGF Chairs - > > The Group Session Request period is now open for GGF17! > > GGF17 will be May 10-12, 2006 in Tokyo, Japan. Please notice that > this is a three-day GGF, instead of our normal four-day event. > > Please go to the following link for Group Session Requesting > instructions and the link to the request form: > http://www.ggf.org/ggf_events_next_chairinfo.htm > > Requests may be updated after initial submission as information (such > as specific agenda) becomes available. Please be aware that the > information supplied on this form is also used for the printed and on- > line schedules. > > The scheduling deadlines are: > > Draft Documents: > - Deadline for submitting working drafts is 31 March, 2006. > Sessions: > - Deadline for session requests for a CG, WG or RG is 17 March, 2006 > - Deadline for AD Hoc/Charter Discussion BOFs is 17 March, 2006. > > The schedule itself will be finalized by the first week in April. > > Thanks, > > Joel > > ------------------------------------------------------------------------ > ----------- > Joel Replogle - Manager of Standards, Global Grid Forum > replogle(a)ggf.org http://www.ggf.org ----- End forwarded message ----- -- "So much time, so little to do..." -- Garfield

1 0

Re: [saga-rg] Phone Call
by Andre Merzky 23 Feb '06

23 Feb '06

Hmm, I thought Thursday is out of option for you - if not, we can easily move it one or two hours before that... Please let me know :-) Others? Cheers, Andre. Quoting [pascal.kleijer(a)knd.biglobe.ne.jp] (Feb 23 2006): > > Hello, > > Ouch, that is a bit late for me. So I will skip it unless I cannot agree with my pillow. > > Good luck! > > > Hi, > > > > there was no more discussion about the phone call - so I the > > liberty to schedule it for today - lets see if anybody can > > make it (sorry Pascal). > > > > Here are the details. > > > > Cheers, Andre. > > > > Date: 23 February 2006 > > Time: 5:00 p.m. Central Europe Time > > Number: 0844.888.8888 (UK) > > 01805.004.102 (Germany) > > +44.870.088.5706 (others) > > Code: 808044 > > > > -- > > "So much time, so little to do..." -- Garfield > > > > -- "So much time, so little to do..." -- Garfield

2 1