From the Rails list:
First, some quick definitions:
- functional testing—Trying to break a web application by shoving crap into the forms and clicking buttons the developers don’t click.
- load testing—Trying to break a web application by pounding the hell out of it with tons of simultaneous connections.
- performance testing—Trying to analyze the performance of a web application so that you can compare it against: a previous release, another application, a base “perfect” case, or a requirement.
The tools for all three:
- httperf—http://www.hpl.hp.com/research/linux/httperf/ My personal favorite since it’s the only one that gives detailed statistics of request, response, connection, transfers, etc. and even separates the headers from the body to spot additional problems. Command line only. Additionally it satisfies my fetish that a tool without a standard deviation is useless. Geoffrey Grosenbach and Zed Shaw have published a paid screencast on using httperf: http://peepcode.com/products/benchmarking-with-httperf
- \TestMaker—http://www.pushtotest.com/ More of a testing tool which uses Jython to let you write up pretty complex test scenarios against a wide range or protocols. It’s kind of difficult to use, but if you need to do deep automated testing then it’s the best out there. It can also do “performance” testing, but what they call “performance” testing is more like load testing.
- Siege—http://www.joedog.org/siege/ This is also a good load testing tool, but it won’t tell you anything useful for performance comparison. Like other tools if it doesn’t have a standard deviation then it’s useless for any real work. Siege also doesn’t tell you useful stuff like actual concurrent connections made (httperf does).
- JMeter—http://jakarta.apache.org/jmeter/ I know people are going to hate me for saying this, but please stay away from JMeter. It has the absolute worst statistics of all the tools, is difficult to use, has bad misleading graphs, complicated terminology, never defines “user”, and it can’t figure out if it is a functional testing, load testing, or performance testing tool.
- Watir—http://wtr.rubyforge.org/ My absolute freaking favorite tool for functional testing. It actually uses the real deal Internet Explorer to do it’s tests, so you know that your site works with the majority of your users. It’s also damn easy to use. I had an intern at work using it in an hour and she’d never used Ruby before. You can also do a bit of load testing by launching a ton of independent test cases in a massive loop. You might be able to squeeze some performance metrics out of it by timing how long a page takes to render, which is actually a nice measure of what the users think of your site, but YMMV. This article describes how to use Watir to collect performance metrics.
- Watij—http://watij.sourceforge.net/ Watij (pronounced wattage) stands for Web Application Testing in Java. Based on the simplicity of Watir and enhanced by the power of Java, Watij automates funtional testing of web applications through the real browser.
- curl—http://curl.haxx.se/ Nothing beats running curl in a massive loop or creating a ton of curl processes to pound a server. You won’t get performance metrics out of it, but it will surely tell you if your system can handle the load. It’s also easy to use for quick tests. When I’m testing a Rails deploy I just hit it with curl to see if it’s working well or not.
- SLAMD —http://www.slamd.com/ Never used this, but heard of this one guy who was this big time UNIX guru in Switzerland who managed to get a SLAMD cluster working one time before he shot himself in misery. :) Also doesn’t produce accurate statistics so performance evaluation is useless, but good for load testing, if you can get it running.
- J\WebUnit—http://jwebunit.sourceforge.net/ Actually, Java has a lot of functional test tools that were crammed into the JUnit framework. Rock on if you like coding your tests in Java.
- webunit—http://mechanicalcat.net/tech/webunit/ Kind of the same thing as JWebUnit but for Python. Python has a few tools that I’ve used in the past, but most of them were pre-ALPHA quality. None of them were good for performance testing, just functional or load testing.
- Grinder – http://grinder.sourceforge.net/ A java framework that uses python (jython) scripting for control of the environment. For load testing custom buildouts, it excels.
A common theme you may notice is that all of these tools have very useless statistics, so they don’t help you compare the performance of systems. It may sound like anal academic nonsense, but without at least a standard deviation or range calculation you’ll never be able to find out if your system is performing consistently under load. My recommendation is to use httperf to check the performance of systems and to make sure that your proposed performance enhancements actually do improve things. Other than that you’re pretty much just at the mercy of whatever is available.
Some other background reading on Performance vs. Load vs. Stress testing theory:
http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress-testing.html
An article on using httperf, autobench and openload to test the performance of a Web application:
http://agiletesting.blogspot.com/2005/04/http-performance-testing-with-httperf.html