[occi-wg] Test suite for OCCI

[Sam Johnston]

Gary,

It's a small world - Anna's from my old university department (where I
studied computer science and worked as a casual academic and NT/Unix
sysadmin for a good few years) and my OS lecturer (among others) is doing
some interesting
things<http://www.ok-labs.com/blog/entry/mobile-virtualization-and-the-cloud-a-pleasing-symmetry/>in
cloud too. It's good to see them putting UNSW on the [cloud computing]
map.

Anyway there was a followup
article<http://www.itnews.com.au/News/153819,more-data-released-on-cloud-stress-tests.aspx>(after
the paper was presented at the conference) in which she nails one of
the main problems with performance metrics in the cloud:

"It is not like the (database benchmarking organisation)
TPCC<http://www.tpc.org/tpcc/>,
which measures the performance of relational databases," she said. "There is
a well-defined set of capabilities and usage scenarios for a relational
database, not so for a cloud service."

I have a feeling PyUnit captures the time it takes to run each test, but if
not I'd say it will be reasonably easy to gather this information if it's
deemed useful. I'd wager the more interesting performance information
relates to the workloads themselves rather than the performance of [what
should be a relatively efficient] API.

Sam

On Mon, Oct 26, 2009 at 4:36 AM, Gary Mazz <garymazzaferro at gmail.com> wrote:

> Good idea to get his process started now.
>
> There has been work done in this area by Anna Liu, Associate Professor at
> University of New South Wales School of Computer Science.
>
> Here is a link to th itnews article covering the work:
>
> http://www.itnews.com.au/News/153451,stress-tests-rain-on-amazons-cloud.aspx#
>
> gary
>
>
>
> Sam Johnston wrote:
>
>> Morning all,
>>
>> Last for tonight I promise... I (well, and a bunch of people much smarter
>> than myself) figure the best way to make sure that implementations are
>> interoperable is to develop a comprehensive test suite from the
>> specification. Accordingly I've been looking at the various options and
>> having written the earlier reference implementation in Python (for Google
>> App Engine) I figure PyUnit is as good a candidate as ever. It helps that
>> virtually all systems have python either preinstalled or at least available
>> for download nowadays and that there's a Google App Engine harness for it
>> (gaeunit.py <http://code.google.com/p/gaeunit/>). This should allow
>> anyone (including end users) to point it at their implementations and test
>> compliance without having to download anything. I say "should" because I'm
>> not sure if GAE supports custom HTTP methods yet (e.g. PUT, OPTIONS and more
>> obscure ones like COPY and MOVE), but I'll soon find out.
>>
>>
>> Before I spend too much more time looking into this I wanted to give you
>> all an opportunity to comment, particularly if you have better ideas? Do any
>> of you think this (and test suites in general) are a particularly good or
>> bad idea?
>>
>> Sam
>>
>> ------------------------------------------------------------------------
>>
>>
>> _______________________________________________
>> occi-wg mailing list
>> occi-wg at ogf.org
>> http://www.ogf.org/mailman/listinfo/occi-wg
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.ogf.org/pipermail/occi-wg/attachments/20091026/f7d107a5/attachment.html