Overview

I've been working on the performance of my Drupal site and thought I'd share my results to date in case they might help someone else. My baseline performance using out-of-the-box settings was around 28 requests per second, and the end result is a rate of 68 requests per second.

The site, www.kidpub.com, is lightly loaded, with between 400 and 600 visits per day for a total of 3,000 to 4000 pageviews per day. The system is a dedicated 1.8GHz Celeron with 500M RAM and a 100M ethernet. It's running CentOS. Apache is 2.0.52. Latest build of Drupal 5 and MySQL 5.

I looked at three main areas for performance tweaking: Apache, MySQL, and PHP. Most of the changes are pretty simple but result in good performance gains.

Benchmarking Apache with ab

Most of my tuning was done in Apache. I found that the default settings for the web server were not appropriate for a small system such as mine; in many cases they were 10x higher than I needed. I used the ab utility to benchmark server performance; for example:

ab -n 500 -c 50 http://www.kidpub.com/latest/

This makes 500 requests for /latest with 50 concurrent users. The output of ab looks like this:

Document Path:          /latest
Document Length:        25166 bytes

Concurrency Level:      50
Time taken for tests:   7.344847 seconds
Complete requests:      500
Failed requests:        0
Write errors:           0
Total transferred:      10614843 bytes
HTML transferred:       10405231 bytes
Requests per second:    68.07 [#/sec] (mean)
Time per request:       734.485 [ms] (mean)
Time per request:       14.690 [ms] (mean, across all concurrent requests)
Transfer rate:          1411.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    7  15.5      0      59
Processing:    44  650 210.2    729    1110
Waiting:        2  636 204.9    711    1109
Total:         50  658 207.2    747    1123

Percentage of the requests served within a certain time (ms)
  50%    747
  66%    779
  75%    792
  80%    797
  90%    867
  95%    895
  98%    936
  99%    997
 100%   1123 (longest request)

This is one of my heavier pages. You can see that 99% of the requests were served in under a second, with a total throughput of about 68 requests per second. My goal was to bring this representative page down from its old value of 5s average response time to closer to 1s.

Apache Changes

Most of the tuning done in Apache was around the number of httpd processes started and kept alive (my server uses the prefork MPM).Here's the prefork section from httpd.conf:

<IfModule prefork.c>
StartServers       3
MinSpareServers    2
MaxSpareServers   3
MaxClients      15
MaxRequestsPerChild 5000
</IfModule>

The big problem for a small server is that the default values for these settings is quite high. For example, MaxClients is the total number of httpd child processes that may be running simultaneously. The default value is 150. The issue for Drupal is that each of those child processes is going to allocate a large chunk of physical memory...I typically see 15M to 20M per process on my server. Twenty httpd processes, each allocating 20M, would consume nearly the entire 500M physical memory and the system would start to swap. Add another twenty and the system will be thrashing.

As a rule of thumb, I allocate 50% of the available physical memory to Apache. So, 250M / 20M per process = 12 httpd processes maximum. I add a few more for headroom under the assumption that the median memory footprint is smaller than 20M. This allows plenty of room for MySQL, PHP, and the operating system.

Based on this stanza in httpd.conf, then, my server will initially start 3 httpd processes (StartServers). Based on load, there should always be at least 2 processes idle and waiting for a request (MinSpareServers) but no more than 3 idle httpd processes (MaxSpareServers). At no time will there be more than 15 httpd processes running (MaxClients).

MaxRequestsPerChild is set relatively low (5000 requests). When a process exceeds MaxRequestsPerChild, it is killed and, if necessary, a new process will be spawned to replace it. I do this to minimize the size of each process...an httpd process will allocate memory if needed, but it doesn't release it. For example, if the very first request served by a process is a big one, say 25M, and the remaining requests are small, say 10M, the proces will hang on to its 25M of memory. By limiting the total number of requests a process can serve, I'll get a new process with a small memory allocation periodically. This has to balanced with the cost of process creation, but the impact is minimal.

KeepAlive

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 2

By default, KeepAlive is turned off. By turning it on (KeepAlive On), we allow a single TCP connection to make multiple requests without dropping the connection. For Drupal this is important, since each page typically has several elements on it, and a single hit on that page might make multiple requests. If KeepAlive is off, each element request will result in a new TCP connection and its associated overhead. The total number of requests available per connection is set by MaxKeepAliveRequests, here set to 100. The total amount of seconds the connection will stay up is set by KeepAliveTimeout, and I've set it to twice the length of my longest average page request.

MySQL

Changes to MySQL configuration cover two areas; caching and connections. These are made in /etc/my.cnf

set-variable=max_connections=15
set-variable=max_user_connections=15
set-variable=query_cache_limit=16M
set-variable=query_cache_size=64M
set-variable=query_cache_type=1

The number of maximum connections to the database defaults to 1. Each httpd process can open a connection, so I've increased the maximum value to the largest number of httpd processes I expect to see (MaxClient), or 15. I've also set aside 64M of memory to hold cached queries; on my site, there are several large queries that change infrequently, and the idea is that these will be served from cache when possible.

You can examine the state of the cache from the mysql command line:

mysql> show status like 'qcache%';
+-------------------------+----------+
| Variable_name           | Value    |
+-------------------------+----------+
| Qcache_free_blocks      | 9990     |
| Qcache_free_memory      | 34431360 |
| Qcache_hits             | 2165383  |
| Qcache_inserts          | 461500   |
| Qcache_lowmem_prunes    | 113692   |
| Qcache_not_cached       | 1894     |
| Qcache_queries_in_cache | 28203    |
| Qcache_total_blocks     | 66628    |
+-------------------------+----------+
8 rows in set (0.00 sec)

Here we can see that there have been 2 million cache hits and that about half of the allocated cache is still available (free_memory). The cache has been flushed (lowmem_prunes), so it might be useful to increase the value of cache_size in my.cnf slightly.

Conclusion

Performance tuning is a bit of art. For my site, these changes had a significant impact, doubling the performance and placing safeguards around resource consumption. Be sure to benchmark the site before you start working on tuning so that you have a baseline value to compare against.

Perry Donham
www.kidpub.com

Comments

N6546R’s picture

I should also mention that along with ab, turning on extended status messages in Apache can be useful for a sanity check. The appropriate directives in httpd.conf are:

ExtendedStatus On

and, in either the main server or a virtual server,

<Location /server-status>
SetHandler server-status
Order deny, allow
Deny from all
Allow from .servername.com
</Location>

When accessed at yourdomain.com/server-status, you'll see a set on information about each process slot, including total requests, the state of each process, and so on. It can be useful to get a quick snapshot of the server, and you also can grab a machine-readable summary that can be used in scripts (http://your.server.name/server-status?auto).

ittec’s picture

You say:

MaxClients is the total number of httpd child processes that may be running simultaneously. The default value is 150. The issue for Drupal is that each of those child processes is going to allocate a large chunk of physical memory...I typically see 15M to 20M per process on my server.

I not understand your explanation. If I have a server with MaxClients setup to 256, 1 GB of Ram. Well, if I do:

ps -ylC httpd --sort:rss

The total is 2094 kb, 2 MB in memory, isn't it? So Apache are using only 2 MB totally. I can't understand how apache uses 15-20 M per process :S

Can someone bring light here?

N6546R’s picture

It really depends on what modules you have enabled in Drupal. If you are running a lightweight site with just a few modules, you might see less than 5M per process. On my community driven site with maybe a dozen additional modules running, 15M to 20M per process isn't uncommon. You really need to take a look at your running processes and see what the total memory footprint is with top or another process reporting tool. The bpttom line is that out of the box you will likely run into performance issues related to memory usage.

Perry
www.kidpub.com

jsmoriss’s picture

A few weeks ago I needed to tune an apache web server for a client, and although the process is straight-forward (see memory used by processes, server memory available, etc.), it's also a bit tedious. I expected to find a ready made application / script to do the number crunching, but my search didn't turn up anything, so I rolled my own. ;-) I figured you might find it useful too. You can find the weblog entry here https://surniaulula.com/2012/11/09/check-apache-httpd-mpm-config-limits/ and the script on Google Code as well.

Enjoy!

js.

tjholowaychuk’s picture

Depends on Apache / PHP modules loaded as well :D
____________________________________________________
Tj Holowaychuk

Vision Media - Victoria BC Web Design
Victoria British Columbia Web Design School

puddyglum’s picture

MaxRequestsPerChild is set relatively low (5000 requests). When a process exceeds MaxRequestsPerChild, it is killed and, if necessary, a new process will be spawned to replace it. I do this to minimize the size of each process...an httpd process will allocate memory if needed, but it doesn't release it.

For us, we reduced it from 4000 to 120 and found a great improvement in available memory. Our site is module heavy, and with many authenticated users as well (so cannot use page cache). Reducing this number helped prevent the rare occurrence of going into swap. It reduced apache memory usage by about 40% total. On another site we have mostly anonymous users and keeping it around 5,000 probably wouldn't be as big of an issue.

MikaT’s picture

The my.cnf file would look like this when using MySQL 5.5.x

[mysqld]
max_connections=15
max_user_connections=15
query_cache_limit=16M
query_cache_size=64M
query_cache_type=1

After editing/creating the my.cnf, you can restart the MySQL with following command

sudo service mysql restart