Nginx: default FastCGI cache levels value may exhaust all inodes

Thanks. The 2:2 will help immensely. That was one of the big problems when I switched to the tmpfs and tried to delete the old disk cache. The system very quickly became unresponsive.

I actually had to write a small script to run every 15 seconds that deleted just one of the 2nd level folders at a time.
It took 11 days to remove the whole disk cache this way.

I know some people put varnish in front of their nginx, but we haven't yet, so we do rely on nginx as a primary cache.

Here's the script I used:
NOTE: Use at your own risk. There's an "rm -rf" in thar. I did test it thoroughly to make sure it didn't eat the world when the cache files were all gone. When you have tested it thoroughly on YOUR system, uncomment the "rm" line.

First, we rename the cache folder so that can be deleted slowly while the live directory is re-created when you restart nginx
(run this as root)

mv /var/lib/nginx/speed /var/lib/nginx/speed-old
service nginx restart

Now, here's the script that deletes one 2nd-level directory at a time:

#!/bin/bash

d=`ls -d /var/lib/nginx/speed-old/*/* |head -n1 |cut -d':' -f1`
if [[ "$d" != "" ]]; then
  echo "removing $d"
  #rm -rf "$d"
else
  echo "doing nothing"
fi

And the crons to run it 4 times a minute (done here between 1AM and 10PM to avoid our backup spike). These output to a log file so you can see what it's doing.

* 1-22 * * * /root/bin/rmspeed >> /root/rmspeed.log
* 1-22 * * * sleep 15; /root/bin/rmspeed >> /root/rmspeed.log
* 1-22 * * * sleep 30; /root/bin/rmspeed >> /root/rmspeed.log
* 1-22 * * * sleep 45; /root/bin/rmspeed >> /root/rmspeed.log

I've also attached a munin graph showing our inode exhaustion. I caught it after a couple of weeks, but at the same rate, we would have run out of inodes in about 7 weeks.

I tried many combinations of ionice, and because our VPS drives are buffered so heavily, ionice still ended up queuing a massive # of deletions until they hit the drive and locked up the machine.

The script I used above should do it slowly enough to not take the system down. Each run only deletes one of the middle directories which could contain up to 256 3rd-level directories. Because of the minimal cache settings, the 3rd level dirs are basically empty.

I've also updated my previous comment to first rename the old cache directory so it can be cleaned up while the new 2:2 cache is created by nginx.