At least initially, we're not planning to use the reporting of the load testing tool itself - that's fine if you are load testing application + hardware, but these are application tests only, so we only need something that can generate load (which must include being able to have a session and submit forms etc.). If it has nice reporting on top, that may be useful later.
Stuff we need:
- You should be able to write and run load tests locally, a bit of setup is OK but not too much.
- either needs a test recorder or a very easy scripting language.
- can build test scripts without any code using their plugin which is supported
- allows real browser testing - could add front end / browsermob style testing on top later
- Has an API for writing tests in different languages other than the Selenium one.
- There is already Selenium -> Simpletest code written.
- can't easily run headless.
- very closely tied to browsermob for load testing, not keen on relying on third party services.
- has a pretty active community that is working on this stuff http://rubyforge.org/pipermail/wtr-development/2011-February/002933.html
- can both drive real browsers and do 'headless' testing.
- Tests are written in Ruby, meh.
- no centrally supported test recorders (although there are some projects that do it, no idea how well they work).
- runs on node.js, there are people in the Drupal community using node already.
- very new.
- no test recorder, but the API looks nice and simple - would rather have a simple API than a nice load test recorder personally.
- only does headless testing (but this is by design, maybe that's OK).
Other things to consider:
JMeter - there's already some generic Drupal load tests written.
http://drupal.org/project/browser - not really in favour of having our on PHP load test runner but it is there.
Possibly others - please add them.
I'm far from decided yet, but am leaning towards zombie.js for simplicity.