One of the sites for which I'm responsible has tanked several times recently. Eventually I became convinced that the problem was a distributed botnet attack -- not necessarily an attempt to take the site down, but rather a side effect of virus-driven referer spam. I'm going to describe the symptoms and the (as yet unproven) strategy I'm using, which others may find useful.
Keep in mind that I'm neither a sysadmin nor a programmer.
The symptoms: After cooking along for many months with a very low load average, suddenly the webserver grinds to a crawl. Web visitors see the "unable to connect to the database" message from Drupal. Load average skyrockets to as high as 80. It's still possible to log in via ssh, but you could brew a pot of coffee while waiting.
Killing httpd, waiting a few minutes, and restarting httpd usually clears up the problem, but in a few cases the backup immediately recurs.
In the server access log, I discover sequences of several hundred of this sort of thing:
221.165.62.200 - - [20/Aug/2006:11:11:30 +0000] "GET /blog/1920 HTTP/1.1" 403 307 "http://www.online-249-poker-kiszka-blada.com/pokeronline-tournaments.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705)"
61.111.193.224 - - [20/Aug/2006:11:11:35 +0000] "GET /blog/1920 HTTP/1.1" 403 307 "http://www.online-249-poker-kiszka-blada.com/pokeronline-tournaments.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705)"