Protect 404/403 Flood Control

Protect 40x Flood Control answers bursts of 403 and 404 responses from one client with a cheap 429 Too Many Requests, using Drupal core's Flood API. Vulnerability scanners and bad bots that probe hundreds of missing paths stop costing you a fully rendered error page per probe.

A vulnerability scanner can send thousands of requests in a few minutes, almost all of them to paths that do not exist. Every one of those misses the page cache, so Drupal renders the full 404 page with all of its blocks and writes a "page not found" log entry. On a small hosting plan that is enough to fill the PHP worker pool and slow the site down for real visitors. CDN rate limiting on lower plans usually cannot count by response code, and path rules only catch the tools you already know about.

For example on a typical site the 404 page could take about 150 ms of CPU to render. An example vulnerability scan we recorded sent around 6,600 requests. If those manage to bypass Cloudflare (e.g. varied IPs across different subnets) than can be about 16 minutes of CPU. With a small plan on a host with low CPU, that plus the heavy influx of "page not found" log entries, can take a good portion of the CPU away from legitimate visitors. With this module enabled using its default settings, about 180 of those requests still get the full 404 page and the rest get the 429 at roughly 40 ms each. That brings the scan down to an estimated 5 minutes of CPU.

Problem solved

Once an IP address has received too many 403s or 404s in a short window, its next 40x responses are replaced with a small plain text 429 for a few minutes. The replacement happens before the error page is rendered and before the log entry is written. Real visitors are never affected by someone else's scan, and even a blocked visitor still gets every normal page: only the error pages are replaced.

Measured on a Standard profile site with Olivero and cold render caches, a rendered 404 took about 19 ms and the 429 about 6 ms. Sites with heavier 404 pages save proportionally more.

Features

• Anonymous 403s and 404s only: counted per client IP address. Logged-in users are never counted. 401, 405 and 410 can be counted too.
• Unrouted 404s by default: paths that match no route are what scanners produce. 404s from a matched route, such as deleted content or a Views pager past the last page, are opt-in.
• Block duration: 30 counted responses in 60 seconds starts a 5 minute block by default. A scanner that keeps going does not get a fresh allowance of rendered error pages every minute.
• Optional IP subnet counter: a second counter per /24 IPv4 or /48 IPv6 range, with its own higher threshold, for scanners that rotate addresses within a range.
• Never cached: the 429 is sent with Cache-Control: no-store, private and Retry-After, so it is never stored by the page cache, the dynamic page cache or a CDN.
• Quiet logs: one warning when a block starts, instead of one entry per probe.
• No permanent bans: blocks expire on their own through the flood table, with nothing to clean up.
• Excluded paths and allowlist: asset paths are excluded by default, so a page with broken images never counts against a real visitor. Allowlist IP addresses or CIDR ranges.
• No cost on normal pages: nothing runs unless a request throws a 40x. A counted 404 adds a few indexed flood queries, and page cache hits are never touched.

Post-Installation

1. Enable the module as usual.
2. Visit Configuration > System > Protect 40x Flood Control (/admin/config/system/protect-40x-flood-control).
3. Check that Your IP address as seen by Drupal shows your own address, not your proxy's or CDN's. If it does not, fix the reverse proxy settings in settings.php first, otherwise every visitor shares one counter.
4. Adjust the threshold, window and block duration if needed, and add your office or monitoring IP addresses to the allowlist.
5. (Optional) Enable Count routed 404 responses if crawlers hammer deleted content or deep pager URLs, and Also count per IP subnet for scanners that rotate addresses.

Requirements

• Drupal 10.3 or later. No other modules are required.
• Correct reverse proxy settings if the site is behind a load balancer, proxy or CDN.

How it compares

This module does not try to make every 404 cheap. It makes abusive clients cheap, and tells them to slow down. Real visitors who follow a broken link still get your themed 404 page with its navigation.

• Fast 404 replaces 404s for everyone with an unthemed static page, optionally checking every request against the router. It is very cheap per request, but every visitor loses the themed page, 403s are not covered, and a scanner can keep going at full speed. They can be used together: Fast 404 makes missing static files, and optionally unknown paths, cheap for everyone, and this module throttles the clients that keep probing what gets through, including 403s. Requests Fast 404 answers itself are not counted.
• Core's built-in fast 404 setting does the same for missing static files such as .txt and .js only, and is on by default.
• Suspect Blocker permanently bans IP addresses with the Ban module after bursts of 403s and 404s. This module sends a temporary 429 instead, with nothing to unban.
• Redirect runs before this module, so redirected legacy URLs are never counted and keep working for a blocked client. Blocked requests are also kept out of the Redirect 404 report.
• CDN and WAF rate limiting remain worthwhile. This module is complementary, and catches the scanners those rules miss.

Related modules

Part of a family of modules for keeping bots and scrapers from overloading Drupal sites, without blocking the good ones:

• Protect Form Flood Control limits how often forms can be submitted.
• Protect Views Flood Control limits how often Views exposed filters can be submitted, and how many filters can be combined.
• Crawl Control stops search engines and simple bots from over-crawling filtered and sorted Views listings.