October 2009 - saga-rg - lists.ogf.org

Face-to-Face Meeting
by Andre Merzky 09 Dec '09

09 Dec '09

[ sorry for cross-posting ] Hi DRMAA group, Hi SAGA group, at OGF27, we discussed a joined 2-day face-to-face meeting between the SAGA and DRMAA groups, with focus on the upcoming DRMAA.v2 specification. We expect a SAGA DRMAA extension to be specified in lockstep -- thus the cross-group participation. At the moment, we plan to have that F2F meeting in Berlin (thanks to Alexander for agreeing to host!). We created a doodle form to poll for possible dates - so for all those who intent to participate in the F2F, please add your availability at: http://www.doodle.com/m4z7fy888yndp7f7 The F2F is open of course, but we may have some space constraints - so please *do* fill the doodle soon, so that we can have a reasonable estimate about participation. Cheers, Andre -- Nothing is ever easy.

2 4

next phone call?
by Thilo Kielmann 18 Nov '09

18 Nov '09

Folks, I am sorry but it looks like I was not prepared to start with the regular calls "right away" after Banff. I am sorry that this was the reason that the VU did not call in... Before we miss even more calls: when will be the next one? This week? Next week? Regards, Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/

2 5

Semantics of 'existing' Attributes?
by Mathijs den Burger 03 Nov '09

03 Nov '09

Hi all, What does the word 'existing' mean in the description of the Attributes interface in GFD.90? For example, the spec says that list_attributes() returns 'existing attribute keys'. Say we do: jd = new job_description() print jd.list_attributes() The C++ Python wrapper will print an empty list. The Java implementation will print all defined job description attributes. So, the C++ impl interprets 'existing' as 'having a value'. Since it does not set any default values (a bug?) it prints an empty list. The Java impl interprets 'existing' as 'being defined', and prints all defined attributes. What would be the correct behavior? regards, Mathijs

2 1

Fwd (peter@troeger.eu): New conference slot -> Decided !
by Andre Merzky 22 Oct '09

22 Oct '09

Hi Folx, the DRMAA call slot is now fixed at the same time as our call, in the alternating weeks. Cheers, Andre. ----- Forwarded message from Peter Tröger <peter(a)troeger.eu> ----- > Date: Thu, 22 Oct 2009 14:45:01 +0200 > From: Peter Tröger <peter(a)troeger.eu> > To: drmaa-wg(a)ogf.org > CC: Andre Merzky <andre(a)merzky.net> > Subject: New conference slot -> Decided ! > > Dear all, > > we finally got it: > > http://www.doodle.com/u6zry5bzuumy63n6 > > Starting from next week, we will have the bi-weekly phone conference on > > Wednesday, 20:00 UTC (http://timeanddate.com/s/1ef7) > > Please note that the daylight saving period in Europe already ends on Oct. > 25th, so 20:00 UTC is effectively 21:00 CET from next week. The Doodle poll > was still calculating in summer time. The winter time switch for the U.S. > central time zone happens in November. > > Best, > Peter. > > > > > ----- End forwarded message ----- -- Nothing is ever easy.

1 0

Python read and write
by Steve Fisher 22 Oct '09

22 Oct '09

I think it is important that the SAGA file read and write operations are as natural and efficient as possible. I have no experience of reading and writing binary data - but I do note that the modules array and struct both appear relevant. I would expect that the operations read and readline and readlines for which I paste the standard doc below: file.read([size]) Read at most size bytes from the file (less if the read hits EOF before obtaining size bytes). If the size argument is negative or omitted, read all data until EOF is reached. The bytes are returned as a string object. An empty string is returned when EOF is encountered immediately. (For certain files, like ttys, it makes sense to continue reading after an EOF is hit.) Note that this method may call the underlying C function fread() more than once in an effort to acquire as close to size bytes as possible. Also note that when in non-blocking mode, less data than was requested may be returned, even if no size parameter was given. Note This function is simply a wrapper for the underlying fread() C function, and will behave the same in corner cases, such as whether the EOF value is cached. file.readline([size]) Read one entire line from the file. A trailing newline character is kept in the string (but may be absent when a file ends with an incomplete line). [6] If the size argument is present and non-negative, it is a maximum byte count (including the trailing newline) and an incomplete line may be returned. An empty string is returned only when EOF is encountered immediately. Note Unlike stdio‘s fgets(), the returned string contains null characters ('\0') if they occurred in the input. file.readlines([sizehint]) Read until EOF using readline() and return a list containing the lines thus read. If the optional sizehint argument is present, instead of reading up to EOF, whole lines totalling approximately sizehint bytes (possibly after rounding up to an internal buffer size) are read. Objects implementing a file-like interface may choose to ignore sizehint if it cannot be implemented, or cannot be implemented efficiently. Supporting these calls would make reading normal text files completely natural. It would appear that it could be used for binary data too in conjunction with the struct module. However I have no experience of handling complex binary structures in Python. Does anybody have practical experience of handling large volumes (at least GB) of binary data in Python? We may well require a totally different call to read into a buffer - but then how do you handle the data once you have it in your buffer other than writing it out again? Steve

2 1

phone call cadence
by Andre Merzky 21 Oct '09

21 Oct '09

Hi all, the topic of reviving our WG phone calls has come up at OGF27 (and before actually). We would like to propose a bi-weekly schedule, at Wednesdays 10pm CET (3pm Central). Note that the DRMAA WG will hopefully pick the same slot, also bi-weekly, so that the calls for both groups will alternate. Is there anybody on the list who would like to participate, but can't make this time slot? Cheers, Andre. -- Nothing is ever easy.

3 6

message api...
by Andre Merzky 21 Oct '09

21 Oct '09

Hi folx, we had a heated discussion about the message API extension - and a lot of confusion about naming. Just stumbled over this piece, and wanted to share: http://stage.vambenepe.com/archives/96 We'll follow up with the message API discussion on one of the next calls. Cheers, Andre. -- Nothing is ever easy.

1 0

Re: [SAGA-RG] Fwd (mathijs@cs.vu.nl): Suboptimal things in SAGA
by Andre Merzky 21 Oct '09

21 Oct '09

Hi Matthijs, we did not manage to discuss all at OGF - agenda was pretty packed already. Thus again by mail... Quoting [Thilo Kielmann] (Oct 15 2009): > > more food... > > ----- Forwarded message from Mathijs den Burger <mathijs(a)cs.vu.nl> ----- > > > Subject: Suboptimal things in SAGA > > From: Mathijs den Burger <mathijs(a)cs.vu.nl> > > To: Thilo Kielmann <kielmann(a)cs.vu.nl> > > > > Hi, > > > > Here's a list of things I feel that are not optimal in SAGA right now. > > Maybe interesting for (lunch)discussions at OGF? > > > > 1. Exception handling in engines with late binding is a pain. When > > multiple adaptors are tried automatically and all fail, it is hard to > > figure out what actually went wrong. JavaGAT also has this problem. > > Users run away screaming when seeing their first nested exception: it > > contains 10 backends they never heard of nor asked for complaining about > > stuff they do not understand. Agree, it is painful. But what can you do? At best, the engine is able to emply some heuristics to extract the most relevant exception and push that to the top level. Your application should then only print that message to stderr, by default. The only real 'solution' would be to disable late binding... Or do you see any other way? > > 2. The any:// scheme is evil. It works well if the available backends > > change a lot, but that is not the case in practice. Users know very well > > what backends are used for what Grid sites: it's rather static info. Well, then they should use the backend specific URLs, not 'any'! 'Any' is exactly for those cases where the backend is *not* known - in all other cases, it does not make sense to use it. If you think this is too much trouble for your users, simply disable 'any'. The spec says: "The SAGA API specifiation allows the use of the placeholder âanyâ (as in any://host.net/tmp/file) A SAGA compliant implementation MAY be able to choose a suitable protocol automatically, but CAN decline the URL with an IncorrectURL exception." > > A much cleaner design (e.g. followed in IbisDeploy) to alleviate > > problems 1 and 2 is to define a number of Grid sites that each have > > certain backends and for which you have certain credentials. A SAGA > > engine can then, per site, only try the adaptors that make sense in the > > first place. Hmm, isn't that what is happening? The default session should contain saga contexts for those backends you have security credentials for. As the adaptors live in that session, only those adaptors should get active for which a context exists. You probably mean that the other adaptors still throw an AuthorizationFailed exception if no context is available? Well, one can disable the adaptors. So, I guess what I try to say is that the saga::session can be used to specify the backends to use, via the contexts. > > This limits the backends tried to the ones explicitly > > specified by the user, which makes it much more comprehensible what is > > going on. It is also faster, since not all adaptors have to be tried. > > Currently, there is no generic way in SAGA to limit which adaptors or > > credentials are used for a site. JavaGAT does have such functionality. The generic way is to create a session with those contexts (aka credentials) attached which you want to use. Say, you want to limit the set of active adaptors to the globus adaptors, do saga::session s; saga::context c ("globus"); s.add_context (c); saga::filesystem::file f (s, url); This should get you only the globus adaptor - all others will bail out, right? (sorry if my answer is a repetition from above) > > 3. Sessions with multiple contexts of the same type should be forbidden. > > Trying them all may have weird and unwanted side-effects (e.g. creating > > files as a different user, or a security lockout because you tried to > > many passwords). It confuses the user. This issue is related to point 2. This is a tough one. The problem here is that a context type is not bound to a backend type. Like, both glite and globus use X509 certs. Both AWS and ssh use openssl keypairs. Both local and ftp use Username/Password, etc. I don't think this is something one can enforce. We had the proposal to have the context types not bound to the backend *technology* (x509), but to the backend *name* (teragrid). This was declined as it makes it difficult to run your stuff on a different deployment using the same cert. > > 4. URL schemes are ill-defined. Right now, knowing which schemes to use > > is implementation-dependent voodoo (e.g. what is the scheme for running > > local jobs? Java SAGA uses 'local://', C++ SAGA used 'fork://'). There > > is no generic way of knowing these schemes other than 'read the > > documentation', which people don't do. Essentially, these schemes create > > an untyped dependency of a SAGA app to a SAGA implementation, causing > > SAGA apps not to be portable across implementations unless they all have > > the same adaptors that recognize the same schemes. Correct. Schema definition is not part of the spec. I argue it should not be either, as that can only be a restrictive specification, which would break use cases, too. Only solution right now is to create a registry - simply a web page which lists recommendations on what scheme to use for what backend. Would that make sens to you? > > 5. Bulk operations are hard to implement and clumsy to use. Better would > > be to include bulk operations directly in the API where they make sense. > > It's much simpler to implement adaptors for that, and much easier for > > users to use and comprehend. Oops - bulk ops were designed to be easy to use! Hmmm... About the hard to implement: true, but iff they are easy to use, then that does not matter (to the SAGA API spec). Why were bulk ops not explicitely added to the spec is obvious: it would (roughly) double the number of calls, and would lead to some pretty complex call signatures: list <list <url> > listings = dir.bulk_list (list <url>); list <int> results = file.bulk_read (list <buffer>, list <sizes>); Further, this would lead to even more complex error semantics (what happens if one op out of a bulk of ops fails?). This all is avoided by the current syntax foreach url in ( list<url> ) { tc.add_task (dir.list <Async> (url)); } tc.wait (All); Not that difficult to use I believe? > > Sorry for the rant :) Hey, thats ok! :-) Also, I am very biased in my answers as you'll notice, and probably somewhat defensive, too. So, it would be nice to hear from others, too! Cheers, and thanks, Andre. -- Nothing is ever easy.

2 3

Re: [SAGA-RG] Fwd (mathijs@cs.vu.nl): Re: speeding up (???) Xterior
by Andre Merzky 18 Oct '09

18 Oct '09

Hi Mathijs, I'll answer on the list if you don't mind, as several items relate to earlier discussions we had... Quoting [Thilo Kielmann] (Oct 15 2009): > > food for thought... > > ----- Forwarded message from Mathijs den Burger <mathijs(a)cs.vu.nl> ----- > > > Subject: Re: speeding up (???) Xterior > > From: Mathijs den Burger <mathijs(a)cs.vu.nl> > > To: Thilo Kielmann <kielmann(a)cs.vu.nl> > > Cc: Tudor Zaharia <tudor.zaharia(a)gmail.com> > > > > On Wed, 2009-10-14 at 22:38 +0200, Thilo Kielmann wrote: > > > > > we just had discussed performance problems with getting file size and > > > permission for all entries of a directory. > > > > > > It seems like there might be a fast solution we may have overlooked so far: > > > the directory (defined in namespace??) has methods for getting file size > > > and permissions with a URL parameter, denoting a file you get via d.list() > > > > > > Andre suggested that d.list may even cache all the infos about its entries > > > such that d.list() would be the only call talking to the backend. > > > > We did not overlook that. The problem is that 'd.getSize(entryURL)' has > > to be called for EACH entry you get from d.list(). An adaptor has > > basically has two options of implementing that: > > > > 1. each getSize(URL) call performs a separate remote operation. That is > > what all adaptors in Java SAGA currently do (e.g. via an FTP or SSH > > command). With large remote directories, this results in a DOS attach of > > the remote server which then shuts you out, or simply becomes > > unreponsive (we see that with our FTP and SSH adaptors) > > > > 2. the adaptor caches all directory information, including file sizes, > > modification dates etc. It would then have to perform only one remote > > operation, and the retrieved info can be reused for subsequent > > getSize(), list(), getLastModificationDate() etc. calls. However, there > > is no mechanism in SAGA to invalidate or update such a cache, and it may > > fill up your memory rather quickly. Also, each adaptor has to > > reimplement caching. This can be circumvented by letting the engine > > perform the caching, but again, there is no general mechanism to > > invalidate or circumvent such a cache. But well, there are standard ways to invalidate/refresh caches, most commonly via a time-to-live for the cache. Even if you set that ttl to only a couple of seconds, you should see exactly the speedup you are looking for. I don't think that this is too complicated, really (pseudo code): file.get_size (url u) { if ( cache.data.empty || cache.created () - time.now () > 5 ) { cache.data = dir.get_sizes (u.get_pwd ()); cache.created = time.now (); } return cache.data [u.get_name ()]; } As for where to cache: engine, adaptor, or some external lib: that is a tradeoff which is best decided in your implementation IMHO. > > The solution would be to have method calls like d.getSize(List<URL> > > entries). The adaptor can than retrieve the files sizes of all entries > > as efficiently as possible. The current way of specifying such bulk > > operations is via TasksContainers, which are very tedious to analyse. That is an implementation problem. I don't think we should expose all kind of calls which are easier/faster to implement on application level. Nobody ever claimed SAGA is easy to implement! ;-) Cheers, Andre. > > Java SAGA does not do that at all (by default, it simply starts a Thread > > for each Task: an even more effective DOS attach for many remote > > directory entries). -- Nothing is ever easy.

1 0

Python SAGA bindings proposal
by Thilo Kielmann 15 Oct '09

15 Oct '09

All, for today's session, I have put (the current snapshot of) our proposed Python language binding online: http://www.cs.vu.nl/~kielmann/pysaga/ Please note that this is a temporary storage. Once the discussion will have settled, I will remove the files again, in favour of a more official and stable location. Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/

1 0