Problem/Motivation

I have recently tried the ASN-based bot-blocking feature, and it works extremely well.

It's exceptionally efficient at blocking non self-identifying, or even cloaked bots, with spoofed user agent strings (see below), trying to pass as a regular visitor. Sadly, the "Limit for crawler / bot traffic (visitors that openly identify as crawlers / bots)" (bot_traffic) method has become less and less useful, due to these bad actors ...

I started with this setting:

$settings['crawler_rate_limit.settings']['regular_traffic_asn'] = [
  'interval' => 30,
  'requests' => 30,

... and when I looked in the logs, it seemed that somewhere in the process, some IPs were not included under an ASN in the database, so that IPs from the same ASN were both getting blocked ("429") and not ("200") at the same time. The ratio was roughly that 75% were blocked while 25% slipped through ...

Example (notice the spoofed user agent):

429
145.223.132.137 - - [16/Apr/2026:10:36:49 +0200] "GET /removed-path HTTP/1.1" 429 2850 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/184.1"

200
145.223.132.140 - - [16/Apr/2026:10:36:49 +0200] "GET /removed-path HTTP/1.1" 200 91430 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"

But when I took a closer look, it turned out that the erroneous 200's happened every 30. second, matching the interval, which makes sense: In effect, the rules needed re-training at that moment and the bots needed to hit the ASN limit.

I then tried tried this:

'interval' => 600,
'requests' => 600,

... which worked well, except that every tenth minute, 600 bot visits were needed for the block to kick in, spiking Load Average to 10 (see "Log data excerpt" below). Tightening up interval/requests ratio to this made all the difference:

  'interval' => 600,
  'requests' => 60,

Load Average is now around a healthy ~0.65, and CPU went down from ~40% to ~8%, due to the serve/block ratio now hitting numbers like 81/384 (i.e. 20% served, 80% blocked), resulting in much less strain on the server:

Crawler Rate Limit ASN efficiency

So perhaps the interval/requests values for the ASN example in the README can be tightened up, to make this method more efficient from the get go?

Steps to reproduce

Proposed resolution

Set the interval/requests values for the ASN example in the README to this?

'interval' => 600,
'requests' => 60,

Remaining tasks

User interface changes

API changes

Data model changes

Log data excerpt

11:00 - 11:45

# for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "/2026:11:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { printf("%5d %5d %5d\n", NR, served, limited) }'; done

Min Total 200   429  <<< columns
00  582   420   162  START 30/30 (interval/requests)
01  605   437   168  miss rate ~25%
02  579   386   193
03  582   415   167
04  561   417   144
05  539   353   186
06  600   417   183
07  563   459   104
08  703   703     0  START 600/600 (interval/requests)
09  701   701     0  10 minute spikes - Load Average (15 min) = ~4 / CPU ~40%
10  659   602    57
11  706   495   211
12  681   493   188
13  664   422   242
14 1149   932   217
15  556   341   215
16  567   350   217
17  598   385   213
18  545   545     0
19  585   585     0
20  555   537    18
21  520   317   203
22  427   278   149
23  483   285   198
24  513   312   201
25  560   356   204
26  522   277   245
27  534   263   271  START 600/60 (interval/requests)
28  539   297   242  Load Average (15 min) = ~0.65 / CPU ~8%
29  537   163   374
30  408   101   307
31  452   100   352
32  474   112   362
33  462   120   342
34  412    63   349
35  433    68   365
36  397    57   340
37  410   142   268
38  633   406   227
39  443    78   365
40  570   231   339
41  400    83   317
42  389    73   316
43  465    81   384
44  484   163   321
45  438   125   313
CommentFileSizeAuthor
Crawler-Rate-Limit-ASN-efficiency.png126.83 KBressa

Issue fork crawler_rate_limit-3585204

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

ressa created an issue. See original summary.

ressa opened merge request !19

ressa’s picture

ressa
he/him
commented
Status: Active » Needs review
ressa’s picture

ressa
he/him
commented

The serve:block ratio has switched from about 2.5:1 to 1:2.5, very nice:

# for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { printf(" %5d %5d %5d\n", NR, served, limited) }'; done
00 35953 25555 10398
01 34722 24527 10195
02 36083 25692 10391
03 32064 21789 10275
04 33539 23478 10061
05 31702 21294 10408
06 30591 20648  9943
07 32933 22107 10826
08 31334 21725  9609
09 35453 24816 10637
10 33092 23517  9575
11 31820 16937 14883
12 27496  7962 19534
13 23466  6363 17103
ressa’s picture

ressa
he/him
commented
Issue summary: View changes
vaish’s picture

vaish commented

@ressa thanks for the detailed post. It nicely explains how to use server logs to arrive at the rate limiter settings that make sense for your particular case. It's impressive to see the reduction in server load and bandwidth you were able to achieve.

My concern is that ASN based rate limiting may affect genuine traffic if set too low. Single ASN covers many IP addresses and for all of those together you are allowing only 60 requests per 10 minutes. What if ASN belongs to a popular ISP in your area? All the visitors using that ISP will be counted towards the single low limit and they may end up being rate limited. Did you consider that possibility?

I'm also curious what's your setting for regular traffic? It would make sense to have lower limit for regular traffic (IP address + user agent) then for ASN. Otherwise, you will always hit ASN limit before regular traffic limit has been reached.

That being said, I agree that numbers used in the README should be tweaked. Unfortunately, there is no way to provide numbers that will work for everyone. Numbers you want to use will depend on the amount of traffic your server receives and on the server's capacity to handle it.

ressa’s picture

ressa
he/him
commented
Title: Stricter interval/requests limits in ASN example? » Drop older ASN registrations gradually via First-In-First-Out (FIFO) queue
Category: Task » Feature request
Issue summary: View changes

You're welcome @vaish, thanks for your thorough reply as always. Showcasing all the helpful log example commands from the README was a nice side effect :)

You're right, those settings do risk blocking genuine traffic. Though in my case, the number of visits are not high enough, so there is no danger of grouping several real visits under a single ASN, blocking real visitors.

These are my settings:

$settings['crawler_rate_limit.settings']['regular_traffic'] = [
  'interval' => 30,
  'requests' => 30,

$settings['crawler_rate_limit.settings']['regular_traffic_asn'] = [
  'interval' => 600,
  'requests' => 60,

Thinking more about this, I do agree with you that the example settings will not work for everyone, they need to be set on a case by case basis ...

And actually, about this issue -- my only problem with the 600/600 setting for ASN is the recurring spikes every ten minutes, where the "quota" is reset to zero and needs to hit 600 visits for an ASN, before 429's kick in again, which puts the CPU under strain.

So instead of tweaking the numbers, 600/600 could actually work fine, if only there was a more gradual transition, and a fraction of the older entries were deleted at some interval. Maybe it's possible to introduce a First-In-First-Out (FIFO) queue, where older registrations are dropped gradually?

Something like this:

  1. The setting is 600/600
  2. The 600 seconds interval is divided into ten sections of 60 seconds
  3. When the 600 seconds have passed, the IP-addresses registered in the first 60 seconds are dropped, and the next 60 seconds are appended

This way, 90% of the counted IP's are retained, and the interval for comparison slowly slides "one minute ahead", step by step. Do you think something like that is possible, and worth considering? Please ask, if my request doesn't make sense :)

vaish’s picture

vaish commented
Title: Drop older ASN registrations gradually via First-In-First-Out (FIFO) queue » Stricter interval/requests limits in ASN example?
Category: Feature request » Task
Status: Needs review » Needs work

Your request makes sense. However, it's out of the scope of Crawler Rate Limit module. Actual rate limiting is performed by third party library nikolaposa/rate-limit. Changes you are proposing would need to be implemented in that library, not in CRL module.

This issue should focus on updating settings example in README file.

ressa’s picture

ressa
he/him
commented
Status: Needs review » Needs work

I understand, and thanks for the feedback and tips on where to suggest this feature. I created Drop older registrations gradually via First-In-First-Out (FIFO) queue #59, as well as an issue here, postponed until or if it gets added, #3590091: Drop older ASN registrations gradually via First-In-First-Out (FIFO) queue.

  • vaish committed 9e69e521 on 3.x authored by ressa 
    docs: #3585204 Update interval/requests values in the settings example...
vaish’s picture

vaish commented
Status: Needs work » Fixed

Merged. I tweaked both regular traffic and asn values. Thanks @ressa.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

ressa’s picture

ressa
he/him
commented

Thanks @vaish, those updates and values look good.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.