Last updated November 28, 2011. Created on December 15, 2009.
Edited by dstol, achton, pillarsdotnet, alexkb. Log in to edit this page.

General documentation about ab can be found at Apache Bench (ab)

Begin by performing a

git pull

in order to ensure your source tree is up-to-date.

Then, execute the following from the command line before applying your new patch. This will establish a baseline from which to judge performance improvements or decreases.

ab -c1 -n500 http://head.localhost/

The -c command specifies the number of concurrent requests and -n specifies the total number of page requests. In 99 out of 100 cases it is not necessary to specify a higher value than 1 for the concurrency. Increasing c will only put more stress on your server. This is useful if you want a general stress test, but it will falsify your results if you only want to test the impact of a particular patch on a particular Drupal page.

This will produce output similar to:

This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done

Server Software:        Apache/2.0.55
Server Hostname:        localhost
Server Port:            80

Document Path:          /head/index.php
Document Length:        16668 bytes

Concurrency Level:      1
Time taken for tests:   36.695602 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      1718500 bytes
HTML transferred:       1666800 bytes
Requests per second:    2.73 [#/sec] (mean)
Time per request:       366.956 [ms] (mean)
Time per request:       366.956 [ms] (mean, across all concurrent requests)
Transfer rate:          45.73 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   316  366  29.7    362     608
Waiting:      298  346  29.3    342     588
Total:        316  366  29.7    362     608

Percentage of the requests served within a certain time (ms)
  50%    362
  66%    367
  75%    370
  80%    373
  90%    384
  95%    390
  98%    462
  99%    608
100%    608 (longest request)

There are two important parts of this result:

1) Requests per second.

This is a good value to judge the overall performance impact. Higher is better.

2) The Connection Times table

The first row of the table specifies the headers for the columns. We are mostly interested in the second (mean) and third ([+/-sd]) column of the last row (Total). The mean or average tells us how long a page request took on average (in milliseconds). Here, lower is better. The value is actually not very different from the Time per request value; it's just that the computation is done "the other way around".

What makes this value interesting is the fact that ab provides the standard deviation of the value in the third column. The standard deviation is a means to judge how accurate a particular test result is. If the standard deviation is large, then this is an indicator that something wasn't working as it should, for example, the webserver was under high load or there was a network lag. If this is the case, you will usually see that in the table at the bottom (percentage of the requests served within a certain time) the last value is much larger than the first one. In this case you should investigate your experimental setup and try to reduce the standard deviation.

Now, apply your patch. Here we're showing that you can retrieve the patch file with curl and apply it with patch

curl -LO http://drupal.org/files/issues/patch_file.patch
git apply -v patch_file.patch

Now re-execute the command above, and compare the results:

This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done

Server Software:        Apache/2.0.55
Server Hostname:        localhost
Server Port:            80

Document Path:          /head/index.php
Document Length:        17100 bytes

Concurrency Level:      1
Time taken for tests:   37.903596 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      1761700 bytes
HTML transferred:       1710000 bytes
<strong>Requests per second:    2.64 [#/sec] (mean)
Time per request:       379.036 [ms] (mean)
Time per request:       379.036 [ms] (mean, across all concurrent requests)
Transfer rate:          45.38 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   318  378  35.5    376     690
Waiting:      301  358  35.4    355     672
Total:        318  378  35.5    376     690

Percentage of the requests served within a certain time (ms)
  50%    376
  66%    381
  75%    385
  80%    388
  90%    396
  95%    401
  98%    436
  99%    690
100%    690 (longest request)

Interpreting results

Ideally, we now have two sets of values from our tests: The mean of the time taken per request and its standard deviation.

Without patch:

366 +- 29.7 ms

With patch:

378 +- 35.5ms

We can see that the second value is higher by 12ms and we might draw the conclusion that the patch makes Drupal slower. However, there is also the standard deviation (which is rather large at almost 10% of the result). If you add the standard deviation of the first result and subtract the standard deviation from the second you will get:

Without patch:

396 ms

With patch:

342 ms

The order is now reversed.

This means the two results are within their respective standard deviations. This means that the test wasn't able to deliver a decisive result. In such a case you might try to increase the number of requests that ab does.

Ideally, your results should be several standard deviations apart to allow for a clear statement on the performance impact of a patch.

Looking for support? Visit the Drupal.org forums, or join #drupal-support in IRC.

Comments

VladSavitsky’s picture

"The value is actually not very different from the requests per second value; it's just that the computation is done "the other way around". "
This values is different. Seems you mean:
"The value is actually not very different from the Time per request value; it's just that the computation is done "the other way around". "

benjifisher’s picture

I have been running ab on my server, and I worried because I was getting almost as many failed requests as complete ones:

Complete requests:      1000
Failed requests:        909
   (Connect: 0, Receive: 0, Length: 909, Exceptions: 0)

According to http://www.celebrazio.net/tech/unix/apache_bench.html there is no need to worry: ab calls it a failure if it fetches the page twice and gets a different number of bytes. I only wish I could confirm this by reading the official docs. (I know, I should read the source!)