Problem/Motivation

Last night, a massive bot horde arrived, and the server was timing out. So this morning I looked around for additional measures, and found Cookie Bot Protection, which works really great with Crawler Rate Limit.

Out of curiosity I checked the HTTP response status codes. Luckily, the two solutions use different codes: Crawler Rate Limit uses 429 and Cookie Bot Protection uses 302 or 401, so they can be easily distinguished.

I added Cookie Bot Protection 11.49 today, and the changes can be seen in the log excerpts below.

As can be seen, Cookie Bot Protection takes over as the most active blocker, maybe simply because "Co" comes before "Cr" in the alphabet?

Since Crawler Rate Limit has a strong focus on low resource usage, I believe it would be ideal, if it was the first line of defence, and Cookie Bot Protection could handle those bots that get through.

I included "Served pages -- 200" to show how efficient the two modules are in combination, accepted requests dropped from ~25% to 1-2%.

Steps to reproduce

Use Crawler Rate Limit with Cookie Bot Protection, and see that Cookie Bot Protection seem to execute first.

Proposed resolution

We could raise the http middleware value to 300, to make sure Crawler Rate Limit is placed first in the order of execution, making it the first defence layer blocking with a 429 Too Many Requests response. Cookie Bot Protection can handle those bots that get through, via cookie inspection.

Remaining tasks

User interface changes

API changes

Data model changes

Log excerpts

Crawler Rate Limit -- 429

$ for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per hour
00  67744  36810  30934     45%
01  68666  36387  32279     47%
02  69533  36399  33134     47%
03  68887  36768  32119     46%
04  67813  35607  32206     47%
05  68561  34977  33584     48%
06  67208  34558  32650     48%
07  69486  33771  35715     51%
08  91217  39016  52201     57%
09  88172  38798  49374     55%
10  90250  39206  51044     56%
11 110584  64810  45774     41%
12 240202 214850  25352     10%
13 242732 217273  25459     10%
14 236387 212920  23467      9%
15 246129 220925  25204     10%
16 248577 223127  25450     10%
17 254610 227044  27566     10%
18 229108 202590  26518     11%
19 229850 203144  26706     11%
20 167366 148426  18940     11%
21      0      0      0      0%
22      0      0      0      0%
23      0      0      0      0%

Cookie Bot Protection -- 302 or 401

$ for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 302 || $9 == 401) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per hour
00  67744  67691     53      0%
01  68666  68663      3      0%
02  69533  69533      0      0%
03  68887  68810     77      0%
04  67813  67811      2      0%
05  68561  68559      2      0%
06  67208  67204      4      0%
07  69486  69485      1      0%
08  91217  91198     19      0%
09  88172  88170      2      0%
10  90250  90226     24      0%
11 110584  81499  29085     26%
12 240202  46486 193716     80%
13 242732  46612 196120     80%
14 236387  43807 192580     81%
15 246129  46206 199923     81%
16 248577  48067 200510     80%
17 254610  50998 203612     79%
18 229108  50676 178432     77%
19 229850  50816 179034     77%
20 171361  38461 132900     77%
21      0      0      0      0%
22      0      0      0      0%
23      0      0      0      0%

Served pages -- 200

$ for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 200) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per hour
00  67744  51922  15822     23%
01  68666  53165  15501     22%
02  69533  53565  15968     22%
03  68887  52471  16416     23%
04  67813  51970  15843     23%
05  68561  53116  15445     22%
06  67208  51886  15322     22%
07  69486  48605  20881     30%
08  91217  67091  24126     26%
09  88172  63374  24798     28%
10  90250  64532  25718     28%
11 110584  88549  22035     19%
12 240202 234700   5502      2%
13 242732 237154   5578      2%
14 236387 231121   5266      2%
15 246129 241172   4957      2%
16 248577 243826   4751      1%
17 254610 249714   4896      1%
18 229108 223288   5820      2%
19 229850 223825   6025      2%
20 173031 168431   4600      2%
21      0      0      0      0%
22      0      0      0      0%
23      0      0      0      0%

Issue fork crawler_rate_limit-3614314

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

ressa created an issue. See original summary.

vaish’s picture

vaish commented

I just checked the source code of Cookie Bot Protection. It's implemented as http middleware, just like Crawler Rate Limit. Module weight has no effect on the order of execution in this case. Order is determined by the priority configured for the http middleware service in the module's services.yml file. Both modules are setting the priority to be higher than page cache. CRL's priority is set to 240, while Cookie Bot Protection chose to set it to 250. Hence, it has higher priority than CRL.

Crawler Rate Limit

  crawler_rate_limit.middleware:
    class: Drupal\crawler_rate_limit\CrawlerRateLimitMiddleware
    arguments: ['@crawler_rate_limit.manager']
    tags:
      - { name: http_middleware, priority: 240 }

Cookie Bot Protection

  cookie_bot_protection.kernel:
    class: Drupal\cookie_bot_protection\CookieBotProtectionMiddleware
    arguments:
      - '@config.factory'
      - '@logger.factory'
    tags:
      # Before page caching (priority 200)
      - { name: http_middleware, priority: 250 }

ressa opened merge request !27

ressa’s picture

ressa
he/him
commented
Status: Active » Needs review

Great digging, thanks for reporting back. I found the corresponding Drupal core file core/lib/Drupal/Core/DependencyInjection/Compiler/StackedKernelPass.php, and the comment says:

 * In general middlewares should not have heavy dependencies. This is especially
 * important for high-priority services which need to run before the internal
 * page cache.
 *
 * An example of a high priority middleware.
 * @code
 * http_middleware.reverse_proxy:
 *   class: Drupal\Core\StackMiddleware\ReverseProxyMiddleware
 *   arguments: ['@settings']
 *   tags:
 *     - { name: http_middleware, priority: 300 }
 * @endcode

... so I set http_middleware, priority to 300 in the MR. Do you that could work? After all, since we want Crawler Rate Limit to run before anything else, it is the definition of a high priority middleware.

ressa’s picture

ressa
he/him
commented
Issue summary: View changes

I had a chance to benchmark the MR. It makes a big positive difference and the server load is reduced ~50%: As first line of defence, Crawler Rate Limit does a lot of checking very quickly, while Cookie Bot Protection makes a more thorough cookie-based inspection of the bots that slip through.

Also, the total number of requests per minute dropped ~65% from ~4,700 to ~1,700 per minute, probably due to the 429 Too Many Requests response from Crawler Rate Limit, which makes some bots back off. Cookie Bot Protection probably does much "repeat blocking" of the same bot, due to its more accepting 302 or 401 responses, making bots behave differently and maybe try again right away?

Code Meaning Who's Responsible Retry?
3xx - Redirection
302 Found (Temporary) API Provider Follow redirect
4xx - Client Errors
401 Unauthorized You No (add auth)
429 Too Many Requests You Yes (with backoff)

From https://apistatuscheck.com/blog/api-error-codes-cheat-sheet

Before and after from the server logs, per hour:

Crawler Rate Limit - 429

# for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per hour
[...]
06 258871 220553  38318     14%
07 278677 236180  42497     15%
08 161925 119007  42918     26%  <<< 08:21: MR applied and caches rebuild
09  93906  52196  41710     44%
10  19440  10742   8698     44%  <<< 10:13: Log data extracted
[...]

Cookie Bot Protection -- 302 or 401

# for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 302 || $9 == 401) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per hour
[...]
06 258871  88693 170178     65%
07 278677  95123 183554     65%
08 161925  96430  65495     40%  <<< 08:21: MR applied and caches rebuild
09  93906  92190   1716      1%
10  18960  18665    295      1%  <<< 10:12: Log data extracted
[...]

Before and after from the server logs, per minute:

Crawler Rate Limit - 429

# for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "2026:08:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per minute
[...]
12   2514   2125    389     15%
13   4742   4070    672     14%
14   5931   5294    637     10%
15   2155   1838    317     14%
16   4782   3891    891     18%
17   7931   6762   1169     14%
18   6362   5329   1033     16%
19   2567   2204    363     14%
20   2611   2189    422     16%
21   3715   2947    768     20%  <<< MR applied and caches rebuild
22   1073    748    325     30%
23   2702   1784    918     33%
24    876    597    279     31%
25   2218    981   1237     55%
26   2594   1313   1281     49%
27   1880   1222    658     35%
28    679    404    275     40%
29    858    530    328     38%
30   1904    882   1022     53%
31   1524    986    538     35%
32    840    497    343     40%
33   3804   2084   1720     45%
34   1222    724    498     40%
35   1795   1037    758     42%
36    746    385    361     48%
37   1877   1158    719     38%
38    749    365    384     51%
39   2202   1392    810     36%
40   1550    669    881     56%
41   3051   1456   1595     52%
42    851    511    340     39%
43   1876   1137    739     39%
44   1980   1114    866     43%
45    738    423    315     42%
46    773    497    276     35%
47   1923   1184    739     38%
48   1948   1284    664     34%
49   2060   1010   1050     50%
50   2700   1354   1346     49%
51    830    458    372     44%
52   1205    687    518     42%
53   2089   1239    850     40%
54   1134    595    539     47%
55    866    507    359     41%
56    521    328    193     37%
57   2959   1706   1253     42%
58   3186   1447   1739     54%
59   1474    713    761     51%

Cookie Bot Protection -- 302 or 401

# for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "2026:08:$i:" /var/log/apache2/access.log | awk '{if ($9 == 302 || $9 == 401) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per minute
[...]
12   2514    876   1638     65%
13   4742   1617   3125     65%
14   5931   2007   3924     66%
15   2155    778   1377     63%
16   4782   1646   3136     65%
17   7931   2682   5249     66%
18   6362   2175   4187     65%
19   2567    915   1652     64%
20   2611    914   1697     64%
21   3715   1489   2226     59%  <<< MR applied and caches rebuild
22   1073   1036     37      3%
23   2702   2677     25      0%
24    876    856     20      2%
25   2218   2208     10      0%
26   2594   2579     15      0%
27   1880   1867     13      0%
28    679    655     24      3%
29    858    804     54      6%
30   1904   1894     10      0%
31   1524   1495     29      1%
32    840    810     30      3%
33   3804   3760     44      1%
34   1222   1213      9      0%
35   1795   1726     69      3%
36    746    737      9      1%
37   1877   1863     14      0%
38    749    738     11      1%
39   2202   2150     52      2%
40   1550   1532     18      1%
41   3051   3039     12      0%
42    851    840     11      1%
43   1876   1846     30      1%
44   1980   1954     26      1%
45    738    703     35      4%
46    773    715     58      7%
47   1923   1903     20      1%
48   1948   1923     25      1%
49   2060   1993     67      3%
50   2700   2684     16      0%
51    830    812     18      2%
52   1205   1165     40      3%
53   2089   2046     43      2%
54   1134   1120     14      1%
55    866    848     18      2%
56    521    506     15      2%
57   2959   2938     21      0%
58   3186   3176     10      0%
59   1474   1375     99      6%
vaish’s picture

vaish commented
Status: Needs review » Needs work

Thanks for testing the MR and providing detailed feedback. Do you have any new findings since your last report? Regarding your MR, I would reduce the priority to make sure it remains lower than reverse proxy middleware which is also set at 300. For CRL, priority of 260 should be enough. That's a minimal change we can make to ensure CRL runs before Cookie Bot Protection.

ressa’s picture

ressa
he/him
commented
Status: Needs work » Needs review

Thanks for the suggestion, I lowered the priority as you suggested. The patch has worked well since I applied it, and there is calm on the server. Most bots are blocked via CRL ASN blocking (403), followed by CRL standard time limit (429), and Cookie Bot Protection is the last check, as can be seen below. It's fairly calm right now.

Two weeks ago -- Cookie Bot handled 2/3 of visits

Cookie Bot Protection -- 302 or 401

# for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; zgrep "2026:14:$i:" /var/log/apache2/access.log.14.gz | awk '{if ($9 == 302 || $9 == 401) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per minute
00   1165    419    746     64%
01    293    111    182     62%
02    685    260    425     62%
03   1188    430    758     63%
04    641    237    404     63%
05    394    152    242     61%
06   1059    373    686     64%
[...]

Crawler Rate Limit - 403 or 429

# for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; zgrep "2026:14:$i:" /var/log/apache2/access.log.14.gz | awk '{if ($9 == 403 || $9 == 429) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per minute
00   1165    822    343     29%
01    293    242     51     17%
02    685    500    185     27%
03   1188    876    312     26%
04    641    477    164     25%
05    394    290    104     26%
06   1059    738    321     30%
[...]

Today, Cookie Bot handles ~5%

Cookie Bot Protection -- 302 or 401

# for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "2026:14:$i:" /var/log/apache2/access.log | awk '{if ($9 == 302 || $9 == 401) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per minute
00    652    639     13      1%
01    513    501     12      2%
02    286    267     19      6%
03    454    437     17      3%
04    476    460     16      3%
05    364    346     18      4%
06    517    507     10      1%
[...]

Crawler Rate Limit - 403 or 429

# for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "2026:14:$i:" /var/log/apache2/access.log | awk '{if ($9 == 403 || $9 == 429) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done # per minute
00    652     92    560     85%
01    513    122    391     76%
02    286    122    164     57%
03    454    111    343     75%
04    476     62    414     86%
05    364     78    286     78%
06    517     58    459     88%
[...]

  • vaish committed 2f08846b on 3.x authored by ressa 
    task: #3614314 Increase priority of the Crawler Rate Limit http...
vaish’s picture

vaish commented
Status: Needs review » Fixed

Merged. Thank you, @ressa.

Something to keep in mind while generating your reports. HTTP response codes 403 and 302 are quite common in Drupal. Some of the requests you are counting towards CRL or Cookie Bot Protection may actually be unrelated. E.g. 403 could be caused by anonymous user attempting to access Drupal's admin interface.

Here is relevant section from CRL's README:

Blocked requests return HTTP code 403 which is also used by Drupal. In order to distinguish requests blocked by Crawler Rate Limit you will also need to look at the response size. CRL response contains only a single word "Blocked." and response size will be very small (likely 28 bytes) while the size of Drupal's regular 403 page will be at least several kilobytes.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

ressa’s picture

ressa
he/him
commented

Perfect, thank you @vaish!

And thanks for clarifying that many of those 403's could be from elsewhere. I had a look in the log files for 403's, and the recent ones all seemed to be around 4K ([...] HTTP/1.1" 403 4191 "-" "Mozilla [...]) so they are probably from a hard coded user agent block in .htaccess I added recently.

In older log files (before the .htaccess user agent block) I see many inserts like [...] HTTP/1.1" 403 259 "-" "Mozilla [...], I guess those could be from Crawler Rate Limit or Cookie Bot Protection.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.