Gary,
It's a small world - Anna's from my old university department (where I studied computer science and worked as a casual academic and NT/Unix sysadmin for a good few years) and my OS lecturer (among others) is doing some
interesting things in cloud too. It's good to see them putting UNSW on the [cloud computing] map.
Anyway there was a
followup article (after the paper was presented at the conference) in which she nails one of the main problems with performance metrics in the cloud:
"It is not like the (database benchmarking organisation) TPCC, which measures the performance of relational databases," she said. "There is a well-defined set of capabilities and usage scenarios for a relational database, not so for a cloud service."
I have a feeling PyUnit captures the time it takes to run each test, but if not I'd say it will be reasonably easy to gather this information if it's deemed useful. I'd wager the more interesting performance information relates to the workloads themselves rather than the performance of [what should be a relatively efficient] API.
Sam
On Mon, Oct 26, 2009 at 4:36 AM, Gary Mazz
<garymazzaferro@gmail.com> wrote:
Good idea to get his process started now.
There has been work done in this area by Anna Liu, Associate Professor at University of New South Wales School of Computer Science.
Here is a link to th itnews article covering the work:
http://www.itnews.com.au/News/153451,stress-tests-rain-on-amazons-cloud.aspx#
gary
Sam Johnston wrote:
Morning all,
Last for tonight I promise... I (well, and a bunch of people much smarter than myself) figure the best way to make sure that implementations are interoperable is to develop a comprehensive test suite from the specification. Accordingly I've been looking at the various options and having written the earlier reference implementation in Python (for Google App Engine) I figure PyUnit is as good a candidate as ever. It helps that virtually all systems have python either preinstalled or at least available for download nowadays and that there's a Google App Engine harness for it (gaeunit.py <http://code.google.com/p/gaeunit/>). This should allow anyone (including end users) to point it at their implementations and test compliance without having to download anything. I say "should" because I'm not sure if GAE supports custom HTTP methods yet (e.g. PUT, OPTIONS and more obscure ones like COPY and MOVE), but I'll soon find out.
Before I spend too much more time looking into this I wanted to give you all an opportunity to comment, particularly if you have better ideas? Do any of you think this (and test suites in general) are a particularly good or bad idea?
Sam
------------------------------------------------------------------------