November 2005 - saga-rg - lists.ogf.org

Queries and minor points
by Graeme Pound 23 Feb '06

23 Feb '06

Please find below a couple of queries, and diverse minor points, which arose during my preparation of a Java implementation of the API. I anticipate that my comments on the semantics of the API will become a little more substantive in due course. Thanks, Graeme ######### Queries - I am little unclear of the purpose and operation of Attribute.getVectorAttribute() and Attribute.setVectorAttribute(). What is returned if a vector attribute is queried with getAttribute()? Why are a vector of values associated with a key required? - contextType does not appear to be very extensible, how would different security models be specified? ######### Minor points - Exceptions are not described in the SIDL interface, however Exception types are listed in long description of the methods (for example ReadOnlyAttribute and NoSuchKey). This information should be in the SIDL description as it is fundamental to the interface. - capitalise ivec, openDirFlags, openFlags? - JobService.runJob() returns Job and stdin, stdout, stderr. However stdin, stdout, stderr are available from Job.getJobInfo()? Can runJob() simply return an object Job? - enum values are not specified by JobState - LogicalFile argument types not specified for addLocation, removeLocation, replicate. These should be 'string' - The enumerations openDirFlags and openFlags exist in the following packages SAGA.Namespace, SAGA.File and SAGA.LogicalFile (although SAGA.Namespace is imported by SAGA.File and SAGA.LogicalFile). Is this duplication required? - the interfaces Stream and StreamServer implements Attributes, however Attributes does not exist. Is this Attribute (perhaps Attributes is a better name for this class)? - File.readP() and File.writeP() define input argument of type 'pattern'. This type is not defined by the SIDL API, is 'pattern' an undefined class or primitive type? - in Stream void getContext (out context info); should read void getContext (out Context info); - strawman_error.txt: "All objects in SAGA implement ErrorHandler...", the SIDL for these objects should indicate that they implement the ErrorHandler interface. The following objects: Directory File Job JobService LogicalDirectory LogicalFile Multiplexer Stream StreamServer Task TaskContainer ... any others ?

4 5

SAGA use case document: comments needed! (this week)
by Thilo Kielmann 13 Dec '05

13 Dec '05

Dear all, the SAGA-RG has its use case document up for public comment, ending oct 30. So far, there are 0 comments! :-( I hereby urge everybody to have a look and comment, even if the comment is just trivially in favour... Thanks for your help, Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/

10 16

Meeting Notes: Messaging API
by Andre Merzky 05 Dec '05

05 Dec '05

Hi, these are notes from the ad-hoc meeting at SC-05 about a message oriented communication API. The API might be considered for inclusion at into the GGF SAGA API spec at some point - for now it is only supposed provide coherent discussion and development in the interested groups. As a reminder, material about saga can be found at: http://wiki.cct.lsu.edu/saga/ Meeting Participants: --------------------- - Jason Leigh (EVL) - Venkatram Vishwanath (EVL) - Andrei Hutany (LSU) - John Shalf (LBNL) - Andre Merzky (VU/LSU) Definition: ----------- Message: chunk of data which is potentially larger than a network package. Several independend sets for property flags have been identified: Reliability Requirements: - - - - - - - - - - - - - - Reliable all messages are received exactly once. If received, messages are complete - Unreliable messages are either received or not. If received, messages are complete - AtLeastOnce optional, as for Reliable, but messages can be received more than once Correctness Requirements: - - - - - - - - - - - - - - ByteErrors received messages MAY contain byte errors - NoByteErrors received messages MUST NOT contain byte errors Ordering Requirement: - - - - - - - - - - - - Ordered messages MUST be received in order - NotOrdered messages MAY be received out of order API considerations: ------------------- - it was felt that a BSD like connection setup is most useful - asynchroneous recieving of complete messages is needed (viz use cases!) - striping/multicasting on application level is not considered for now (multiple senders/receivers) API proposal: ------------- - establish connection: - bsd like: listen/accept/connect - port range should be specifiable - properties should be specified as flags (changable at runtime) - write: write (buffer, size, BLOCK | NO_BLOCK) message is written completely or not at all (if possible) - read: two step mechanism: (int handle, int size) = query_size (); (char* buffer) = read (handle, size); - handle can be zero, if size is known (one step read) - buffer needs to be allocated by application. - if size is zero, the buffer is allocated by the implementation and returned to be freed by the application (one step read) - if size is smaller then the real message size, message gets truncated, remainder gets lost (read again not possible) - if size is larger, buffer gets patted by 0 - asynchroneous method calls: - as in the SAGA task model, with callbacks - connection shutdown: - as in bsd (close ()) Please feel free to send corrections, comments etc. Thanks, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

5 10

Re: [saga-rg] Task model...
by Andre Merzky 04 Dec '05

04 Dec '05

Hi Pascal, there have been more models in discussion in Boston, true enough. However, I can't really figure out what model you refer to. Some example code would be helpful :-) Cheers, Andre. Quoting [pascal.kleijer(a)knd.biglobe.ne.jp] (Nov 03 2005): > > Hello all, > > Lets enter the arena if it is not too late, sorry for that. > I do agree with many things that have been discussed here. > Being myself a POP (pattern oriented programming) and OOP > (Object Oriented programming) guy I tend to favor nicely > encapsulated models. > > As mentioned by Tom, the template is possible in Java as > well. But that forces the usage of Java 5.0 (aka 1.5) at > least. This isn't recommended since many programs > still run with 1.4. Using them would probably create a very > different binding in Java, C++ or Python, which would end up > in 2 different SAGA implementations/usage, one for OO > languages and one for old fashioned procedurals. > > As to exposing the different implementation to the user I > think this could be avoided in OO languages, or at least it > should not be as obvious as in 4a2. During the GGF meeting > their was a last minute discussion about a `g' solution > which is pretty much pure OO design with a simple method > call whatever model is used. This is I think a good choice > for an OO point of view. You can derivate an abstract class > to specialized classes with synchronous or asynchronous > implementation. This would even open the room for sub > classing it to add custom code for debugging for example or > other treatment without affecting the initial code. > Depending on how the programming is tackled factories aren't > necessary. The problem might come when implementation is > made in procedural languages, it might be necessary to add > additional parameters to the constructor or the calls. > > I don't have the time this morning to make a sample > code. But I hope you got the idea. > > Best Regards, > Pascal Kleijer > > > > > My vote is for 4a1. (Which was option D in our original discussion I > > believe.) > > > > Cheers, > > > > Tom > > > > On Thu, 3 Nov 2005, Andre Merzky wrote: > > > > > > > > Hi John, Thilo, Group, > > > > > > Hartmut and I would like to argue about the C++ bindings. > > > We would like to come to an agreement soon, as our > > > implementation is dealing with the async part right now. > > > > > > As reminder: > > > > > > --------------------------------------------------------------------- > > > Example 4a: more versions in C++ > > > > > > 4a1: d.mkdir ("test/"); > > > d.mkdir_sync ("test/"); // same > > > saga::task t_1 = d.mkdir_async ("test/"); > > > saga::task t_2 = d.mkdir_task ("test/"); > > > > > > 4a2: d.mkdir ("test/"); > > > d.sync .mkdir ("test/"); // same > > > saga::task t_1 = d.async.mkdir ("test/"); > > > saga::task t_2 = d.task .mkdir ("test/"); > > > > > > 4a3: d.mkdir ("test/"); > > > d.mkdir <sync> ("test/"); // same > > > saga::task t_1 = d.mkdir <async> ("test/"); > > > saga::task t_2 = d.mkdir <task> ("test/"); > > > --------------------------------------------------------------------- > > > > > > > > > Quoting [John Shalf] (Nov 02 2005): > > >> > > >>> Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!) > > >> > > >> I prefer 4a1 because it is more readable and the implementation would > > >> be quite straightforward. > > > > > > You favour 4a1. We think that implementation is very straight > > > forward for all three versions. Readability seems not much > > > different to me, and might largely be a matter of taste. > > > > > > > > >> It is also a familiar paradigm for any MPI > > >> programmers and anyone who has played with various proprietary Async > > >> I/O implementations. (its a very familiar and conventional approach) > > > > > > Well, these are C-API's. Is there a C++ binding for MPI, > > > and does it look the same? > > > > > > > > >> I kind of like 4a2 as well from the standpoint of a C++ programmer > > >> (even with Andre's syntax corrections). However, the resulting > > >> bindings will not be very consistent with the approach we would take > > >> for Fortran or C bindings (eg. those would likely look more like > > >> 4a1). > > > > > > But, well, that is the idea of the language binding! I agree > > > that C and Fortran would look more like 4a1, of course. But that > > > is no reason that C++ should look like that as well, or Java, > > > Perl etc. > > > > > > > > >> It is not really much more readable than 4a1. Therefore, I'm > > >> uncertain if it is worth fragmenting our approach to bindings in > > >> different languages when there is not a clear benefit in terms of > > >> readability or implementation complexity. > > > > > > I think that 4a2/4a3 actually allow nicer implementations, as it > > > allows to have the different async parts somewhat separate from > > > the sync parts. We think its nicer :-) > > > > > > > > >> I do a lot of C++ programming, but I find the 4a3 option a bit > > >> obscure both in terms of readability and any advantages it might > > >> confer in terms of implementation. > > > > > > Hehe - I thought the same :-) Hartmut likes that version very > > > much. To me it appealed after pondering over it for a couple of > > > days. Now I think it is cute, and quite expressive. > > > > > > > > >> It would certainly be easier to > > >> create multi-language bindings for a single code base if we stick > > >> with something more conventional like 4a1. > > > > > > I think that C and Fortran bindings for this part are straight > > > forward anyway, there is no need to reflect that in C++... > > > > > > > > >> Each approach is equally readable to me (less so for 4a3). I'm > > >> certainly open to any additional information on how the 4a2 and 4a3 > > >> approaches could simplify implementation. > > > > > > The main point really is that the object itself has only the sync > > > method calls, and the async calls can be put into separate > > > header/sources easily, and build (by default) on top of the sync > > > calls. Of course, you can do that with all three versions, but > > > its not as obvious in 4a1. > > > > > > > > >> If the other approaches > > >> can offer some implementation benefits, then maybe I'd give them > > >> extra consideration, but otherwise, I would prefer a more > > >> conventional approach like 4a1. > > > > > > I would vote for 4a2 or 4a3 (in that order), but 4a1 would be ok > > > if the majority likes that most, of course. Basically its > > > a matter of taste I think. I am happy that the general point > > > seems accaptable to all so far: having sync, async, and task > > > versions of the calls, w/o explicit task factory. > > > > > >> The only implementation I'm outright > > >> against is the 4b example. > > > > > > Good! :-) > > > > > > Thanks, Andre. > > > > > > > > >> -john > > > > > > > > > > > > > > > > -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

5 9

Update on the DEISA SAGA-based tool
by Terry Sloan 24 Nov '05

24 Nov '05

Hi all, Just thought you might be interested to know the JRA7 team from the DEISA project have just released the latest version of their command line tool that is based on parts of NSDir and JobManagement. You can download the software http://forge.nesc.ac.uk/projects/deisa-jra7/ under Latest file releases. Regards, Terry Sloan -- ---------------------------------------------------------------------- Terry Sloan Email: t.sloan(a)epcc.ed.ac.uk EPCC Phone: +44 131 650 5155 WWW : http://www.epcc.ed.ac.uk/ ----------------------------------------------------------------------

1 0

Re: [saga-rg] SAGA conference call
by Andre Merzky 24 Nov '05

24 Nov '05

Are you having fun :-) Well, sad story is that both Shantenu and I are abroad - and we should haver checked in advance if international calls are possible for us - it seems not... :-( I still try to master my phone - but as the number of dial combinations untried is going down rapidly, chances for me joining in are getting slim... - sorry :-( A. Quoting [pascal.kleijer(a)knd.biglobe.ne.jp] (Nov 24 2005): > Date: Thu, 24 Nov 2005 23:10:19 +0900 (JST) > From: pascal.kleijer(a)knd.biglobe.ne.jp > To: goodale(a)cct.lsu.edu, andre(a)merzky.net > Cc: saga-rg(a)ggf.org > Subject: Re: [saga-rg] SAGA conference call > > Any online? Or is this just between me and Steven? > > > > > It's often easier to deal with if it is inline text, rather than an > > attachment; it's certainly easier to reply to issues on the mailing list > > that way. > > > > Cheers, > > > > Tom > > > > On Thu, 24 Nov 2005, Andre Merzky wrote: > > > > > Well well, the iisue list did not feel like attaching > > > yesterday, so lets see how it works out this time... > > > > > > Cu soon, > > > > > > Andre. > > > > > > Quoting [Andre Merzky] (Nov 23 2005): > > >> > > >> Hi all, > > >> > > >> I scheduled the call for tomorrow, here are the details: > > >> > > >> Date: 24 November 2005 > > >> Time: 3:00pm CET > > >> Number: 0844.888.8888 (UK) > > >> 01805.004.102 (Germany) > > >> +44.870.088.5706 (Others ) > > >> Code: 808044 > > >> > > >> This should be: > > >> > > >> +--------------+-------------+-------------+-------------+-------------+ > > >> | Tokyo London | London | Amsterdam | New York | New Orleans | > > >> +--------------+-------------+-------------+-------------+-------------+ > > >> | Thu 11:00 PM | Thu 2:00 PM | Thu 3:00 PM | Thu 9:00 AM | Thu 8:00 AM | > > >> +--------------+-------------+-------------+-------------+-------------+ > > >> > > >> As for an agenda, I propose we start iteration of the issue > > >> list again - I attach the current version. > > >> > > >> Cheers, Andre. > > >> > > >> > > >> > > >> Quoting [Tom Goodale] (Nov 21 2005): > > >>> > > >>> Hi, > > >>> > > >>> does anyone wish to resume having regular SAGA conference calls ? > > >>> > > >>> Cheers, > > >>> > > >>> Tom > > > > > > > -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

2 1

SAGA conference call
by Tom Goodale 24 Nov '05

24 Nov '05

Hi, does anyone wish to resume having regular SAGA conference calls ? Cheers, Tom

4 13

Java implementation issues
by Graeme Pound 23 Nov '05

23 Nov '05

I have been looking at the Strawman SAGA API (v0.2+) with a view to developing a Java implementation of the API over multiple types of resource. Ultimately I am looking to develop SAGA implementations in Matlab and Jython as part of the GeodiseLab activity for the OMII (www.omii.ac.uk) I have some comments about the current SAGA API as it is defined in SIDL from the Java perspective, and some questions about the SAGA approach. There are also some trivial points about the SIDL interface. My key concern is whether the SAGA API (as defined in the SIDL interface in the strawman document) will produce 'nice' Java that will be simple to use. The problem stems from the use of the SIDL keywords 'out' and 'inout' to define method arguments that are passed by reference. This is perfectly acceptable in FORTRAN, C and C++, however when translated to Java the result is clumsy. In Java arguments are (typically) passed by value. In particular primitive types, such as 'long', and immutable types, such as 'String', are effectively passed by value. This URL describes how Java arguments are passed: http://www.yoda.arachsys.com/java/passing.html. When the Babel tool is used to parse the following SIDL definition into Java: package Hello version 1.0 { class World { void getMsg(out int ppp); } } The integer argument is placed within a mutable object that may be altered from within the 'getMsg' method. This solution may be opaque to may Java users of the API. In most situations in the SAGA API passing a single output argument by reference can be avoided by specifying a return argument (where it is currently void). For example: interface JobService { void submitJob (in JobDefinition jobDef, out Job job); ... } becomes: interface JobService { Job submitJob (in JobDefinition jobDef); ... } However, the problem is slightly more complex where multiple output arguments are returned by reference. For example: class File { void read (in long len_in, out string buffer, out long len_out ); ... } In this situation there are many alternatives to passing multiple output arguments by reference. For example; placing the output information in a single object, or, where appropriate, making the variables available as attributes of the class. A similar problem exists in Matlab and Python where arguments are passed by value. Incidentally, here multiple output arguments are supported by returning a vector (or tuple) of variables. The 'pass by value' issue leads me to a more general question about the SAGA API. Is it your objective that the API is strictly consistent between languages? My concern, if this was the case, is that this may lead to a 'worst-of-all-worlds' solution rather than an API that is appropriate for each language. Certainly this issue would arise for object-oriented versus procedural languages. Andre suggested to me that the language bindings were yet to be standardised. Does the scope exist to vary the language bindings to play to the strengths of each target language? Thanks, Graeme Pound Minor points - ExceptionCategory does not have enum values specified - SAGA.Task.Task.throw(); "throw" is a reserved word in Java - Should TaskContainer, File and LogicalFile be a classes or an interfaces? - Enums are frequently not capitalised, stylistically it would be nice to be consistent about this (from the Java POV these would be classes so should be capitalized) - void return type is frequently not specified

5 7

SC-05 meeting notes
by Andre Merzky 22 Nov '05

22 Nov '05

Hi all, as you know, we had a F2F meeting at SC05. Well, it turned out to be a rather small group, but we had a nice chat ;-) Here are the meeting notes: Participants: Thilo Kielmann Shantenu Jha Yusuke Tanimura Andre Merzky - Charter: We has a discussion with our ADs earlier that day (and the day before. And the days before that... ;-). It seems they are to support our WG charter proposal, but asked us to add: - exit strategy - a milestone to revise the API spec process and the document structure. We did that, the new charter draft (with only these changes) is attached. Please comment this week, otherwise the charter will be submitted to the GFSG as is. - Task model: We agreed that, in order to shorten the discussion, and to keep the OO spec free from language issues, the task model will be described in verbose text in the spec, and the explicit incarnation will be left to the language bindings. We further agreed, that for C++ we are going to implement one or more of the models discussed lately, and to get feedback based on the implementation, and to revise the language binding in respect to that feedback. Otherwise we see the group discussing for quite some time still, while the implementation is stuck at that point... - Issue list: we went through the API issue list. I don't have detailed notes on that, but a check of the CVS logs tells you what we did... As a reminder, the CVS details are on the wiki. - We had backroom talks with the Ninf and GridSolve people at SC, about GridRPC. The direction is still the same: work on including the GridRPC API into the SAGA look&feel. However, the GridRPC API might get revised in that process... - we talked to Keith Jackson, and he agreed on reviewing the python language bindings for the API as soon as we get to that stage (after the C++ reference I guess) - we had an interesting meeting about a message based communication API. That is currently not related to SAGA, but could be interesting in the future. Notes about that meeting will follow to this list, in about a week. Comments/questions/corrections are as always welcome! Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

Fwd (mmm2a@virginia.edu): [byteio-wg] ByteIO/Basic Directory/BES Implementation
by Andre Merzky 17 Nov '05

17 Nov '05

FYI, Andre. ----- Forwarded message from Mark Morgan <mmm2a(a)virginia.edu> ----- > From: Mark Morgan <mmm2a(a)virginia.edu> > To: byteio-wg(a)ggf.org, ogsa-bes-wg(a)ggf.org > Subject: [byteio-wg] ByteIO/Basic Directory/BES Implementation > Date: Thu, 17 Nov 2005 10:40:23 -0500 > > Folks, > > I have uploaded to both the BES and ByteIO gridforge sites a zip file > containing a basic prototype implementation of both client and server sides > of BES, ByteIO, and a basic directory protocol. I spent the last week > implementing the BES stuff and the ByteIO and basic directory stuff was > written previously. Seeing as how naming is so central to both ByteIO and > BES, they have all been packaged together. This package uses WSRF.NET to > implement all of the web service stuff and overall is implemented in C#. > Feel free to browse if you'd like. > > -Mark -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

2 2