November 2009 - saga-rg - lists.ogf.org

Face-to-Face Meeting
by Andre Merzky 09 Dec '09

09 Dec '09

[ sorry for cross-posting ] Hi DRMAA group, Hi SAGA group, at OGF27, we discussed a joined 2-day face-to-face meeting between the SAGA and DRMAA groups, with focus on the upcoming DRMAA.v2 specification. We expect a SAGA DRMAA extension to be specified in lockstep -- thus the cross-group participation. At the moment, we plan to have that F2F meeting in Berlin (thanks to Alexander for agreeing to host!). We created a doodle form to poll for possible dates - so for all those who intent to participate in the F2F, please add your availability at: http://www.doodle.com/m4z7fy888yndp7f7 The F2F is open of course, but we may have some space constraints - so please *do* fill the doodle soon, so that we can have a reasonable estimate about participation. Cheers, Andre -- Nothing is ever easy.

2 4

Re: [SAGA-RG] Timeout exception in SAGA?
by Andre Merzky 19 Nov '09

19 Nov '09

Hi Mathijs, I put that on the list, maybe others have feedback. Summary so far: You see an inconsistency: - methods which have a timeout value usually don't throw on a timeout (like: wait simply returns after timeout) - two exceptions so far: - stream::server.serve (timeout) throws Timeout if no client connected while waiting - close(timeout)/cancel(timeout) calls throw NoSuccess after that time if resources could not be released My opinion: Given that the close/cancel timeouts are less frequently used, those exceptions should probably be changed to TimeOut, to be in sync with stream::server.serve(). I don't think we can get rid off with the exceptions for the three listed calls altogether, at least not without changing semantics. Cheers Andre. Quoting [Mathijs den Burger] (Nov 19 2009): > Subject: Re: Timeout exception in SAGA? > From: Mathijs den Burger <mathijs(a)cs.vu.nl> > To: Andre Merzky <andre(a)merzky.net> > Date: Thu, 19 Nov 2009 13:13:41 +0100 > > On Wed, 2009-11-18 at 21:50 +0100, Andre Merzky wrote: > > Quoting [Mathijs den Burger] (Nov 18 2009): > > > On Tue, 2009-11-17 at 23:36 +0100, Andre Merzky wrote: > > > > Hi Mathijs, > > > > > > > > Quoting [Mathijs den Burger] (Nov 17 2009): > > > > > > > > > > Hi Andre, > > > > > > > > > > Page 23 of GFD.90-plus-errata says (in the section on "Freeing of > > > > > resource and garbage collections"): > > > > > > > > > > "Methods whose semantics depend on successful or unsuccessful de-allo- > > > > > cation of resources (such as task.cancel() or file.close()) allow for > > > > > an optional float argument, which defines a timeout for this operation > > > > > (see Section 2.6.3). If resource de-allocation does not succeed within > > > > > this timeout period, a NoSuccess exception MUST be thrown." > > > > > > > > > > Shouldn't that be a 'Timeout' exception instead? > > > > > > > > The 'Timeout' exception usually implies that communication with a > > > > backend is delayed, and that there is a chance that the operation > > > > may succeed later on (see description of TimeOut exception). This > > > > does also hold for close() and cancel(), which can throw TimeOut in > > > > fact, and can be retried. This exception is independent from the > > > > timeout parameter though! All calls which *do* have a timeout > > > > parameter do *not* throw TimeOut when that time has passed, but > > > > simply return (no error). > > > > > > But the notes of the stream_service.close() call say: > > > > > > "if no client connects within the specified timeout, a > > > ???Timeout??? exception is thrown." > > > > > > I'm confused... :) > > > > Me too! ;-) Let me check that stuff again, will answer later... > > Do you have time to reply soonish? > > Reason: we're current writing an XtreemOS document that includes a SAGA > extension package in which we also have an object with a serve() call. > I'd like it to mimic stream_service.serve(), but I'm hesistating because > it seems inconsistent with the rest of the spec. > > The problem is that the description of the Timeout exception does not > prescribe what should be the return value of a non-void method when the > timeout value is exceeded. The method stream_service.serve() could, for > example, return NULL. Still, it seems that throwing a Timeout exception > is more logical. > > -Mathijs > > -- Nothing is ever easy.

1 0

next phone call?
by Thilo Kielmann 18 Nov '09

18 Nov '09

Folks, I am sorry but it looks like I was not prepared to start with the regular calls "right away" after Banff. I am sorry that this was the reason that the VU did not call in... Before we miss even more calls: when will be the next one? This week? Next week? Regards, Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/

2 5

Python bindings: Buffer class issue
by Manuel Franceschini 18 Nov '09

18 Nov '09

Hi all, Quick summary from GFD.90: the SAGA I/O Buffer encapsulates a sequence of bytes to be used for I/O operations, e.g. read()/write() on files and streams, and call() on rpc instances. The recent removal of the buffer class from the Python bindings of the C++ SAGA implementation led us to think again about this issue. The GFD is C/C++ oriented and therefore the Python implementation is all but clear in this regard. Given that that memory management is automatic in Python, the notion of application-managed and implementation-managed Buffer disappears. There is no need for a Python SAGA user to tell the bindings who manages the Buffer, since it is managed by the underlying Python VM. Another more critical issue is the data type used to hold binary data in Python. In Python 2.x the immutable 'str' type is used whereas Python 3.x has a newly introduced immutable 'bytes' type. Let's forget about 3.x for a moment, since 2.x will be around for at least a couple of more years. In order to manipulate large binary datasets, the mmap class [0] could be used, which basically transforms a immutable 'str' into a mutable mmap object. In other words it provides the ability to efficiently modify binary data. In the VU Python bindings the buffer class is still present, while, as previously said, in the C++ Python bindings it was removed recently. I do not see any issues with the removal of the Buffer class in the Python bindings. However, I'm not sure whether I am forgetting some corner cases (e.g. async) that would require a dedicated Buffer class. When removing the Buffer class, the user would simply deal with 'str' type data to pass data back and forth to a SAGA file, stream or rpc. Now, I identified the following crucial questions: 1) Can the Buffer class be safely removed from the Python bindings? 2) Is handling of large binary datasets a primary concern? If yes, how to handle them? 3) Is compliance to Python 3.x a concern right now? In other words, is the eventual migration to 3.x to take into consideration? Cheers, /Manuel [0] http://docs.python.org/library/mmap.html

3 7

Re: [SAGA-RG] SAGA attributes as Python dictionaries?
by Mathijs den Burger 11 Nov '09

11 Nov '09

On Wed, 2009-11-04 at 19:18 -0600, Hartmut Kaiser wrote: > > > On Wed, Nov 4, 2009 at 6:34 PM, Steve Fisher > > <dr.s.m.fisher(a)gmail.com> wrote: > > > > Just a note to remind people that the dictionary syntax in python > > is > > > > just a convenience - you need to implement __setitem__, > > __getitem__, > > > > __delitem__ etc. Our __setitem__ can reject requests to set > > fields > > > > whoch don't already exist for those cases where the set of > > attributes > > > > is not extensible. It just needs a private method called by the > > > > implementation to set the attributes with any defalt values and > > > > indicate whether or not it is extensible. I think it is best to try > > it > > > > and see if there are any practical problems. Also see: > > > > http://docs.python.org/library/userdict.html#module-UserDict which > > may > > > > be useful. > > > > > > It is certainly doable from the technical point of view. My question > > > is what would be the advantages of having a dict-like interface? > > Would > > > that be in this form then: > > > > If I rember the discussion from Banff correctly, the proposal was > > more like > > > > jobDescription.attributes["Executable"] = "/bin/hostname" > > Why not: > > jobDescription["Executable"] = "/bin/hostname" Yes, I'd also prefer this. Much shorter, and it resembles that a job description IS-A Attributes. > > to decouple the attributes from the object properties. The > > dictionary seems to offer a reasonable interface for atributes, so > > why not use it? > > > > not sure if one can then argue why not to use > > > > jobDescription.attributes.Executable = "/bin/hostname" > > Not good as we have no way to ensure only proper attributes are accessed. > For instance > > jobDescription.attributes.Executable1 = "/bin/hostname" > > will go unnoticed as Python simply creates a member Executable1 assigning > the string. Ah, I did not know this! Hmm, that would indeed be a very good reason not to use Python properties for attributes. > > One question about both approaches though, and to also about both > > approaches you list below: how would async ops be realized? > > Async ops are separate and have to follow the get_attribute/set_attribute > interface (which should be implemented in addition anyways) ... which gives more than one way to set/get attributes. I'm not sure what is more 'Pythonic': providing exactly one way to access attributes (i.e. via set_/get_attribute), or to provide a language-specific construct (dictionaries) as well. Technically, providing a dict interface as well seems entirely possible. It seems more a matter of taste whether to use them or not, especially since SAGA attributes are more restrictive than a normal dict. best, Mathijs

6 15

SAGA attributes as Python dictionaries?
by Mathijs den Burger 05 Nov '09

05 Nov '09

Hi all, For the Python language bindings, we (at the VU) are thinking about CCT's idea to use the Python 'dict' interface for SAGA attributes. While we like the idea to use Python-specific constructs in the language bindings, it seems that SAGA attributes differ in semantics from the 'associative arrays' that are Python's dictionaries. Some examples where Python dicts and SAGA attributes differ: 1. They throw different exceptions (e.g. KeyError vs BadParameter / PermissionDenied / DoesNotExist / AuthenticationFailed / AuthorizationFailed / BadParameter / IncorrectState / NoSuccess) 2. Removing SAGA attributes is permanent (AFAIK when talking to Ceriel), while a deleted dict item can be re-added latter 3. Dictionaries accept any hashable object, whereas SAGA attributes only accept a set of defined values. Exposing SAGA attributes as a dictionary may therefore lead to confusion, as the two are similar but not identical. Another option for easy access (which we use in our implementation on top of Java SAGA) is to use Python's 'property' construct for each defined attribute. This allows you to do, for example: >>> from saga import job >>> js = job.JobDescription() >>> js.Executable = "/bin/date" >>> js.Arguments = [ "-R" ] Python translates such property access to calling getter an setter functions that can do additional checking. Very flexible, and the syntax is even shorter than using the 'dict' interface. Wouldn't that be a better option for easy access to SAGA attributes in Python? best, Mathijs

7 12

call...
by Andre Merzky 04 Nov '09

04 Nov '09

Sorry for the botched phone call setup - my area had a power outage. Maybe I leave too far abroad after all ;-) Did the call happen? Any notes? Thanks, Andre. -- Nothing is ever easy.

2 2

Semantics of 'existing' Attributes?
by Mathijs den Burger 03 Nov '09

03 Nov '09

Hi all, What does the word 'existing' mean in the description of the Attributes interface in GFD.90? For example, the spec says that list_attributes() returns 'existing attribute keys'. Say we do: jd = new job_description() print jd.list_attributes() The C++ Python wrapper will print an empty list. The Java implementation will print all defined job description attributes. So, the C++ impl interprets 'existing' as 'having a value'. Since it does not set any default values (a bug?) it prints an empty list. The Java impl interprets 'existing' as 'being defined', and prints all defined attributes. What would be the correct behavior? regards, Mathijs

2 1