The log examples in the README are great and offer many examples to serve as a starting point for building queries that suit user's particular needs.
Instead of adding more examples to README, we could create a documentation page for this purpose, where more examples could be added without making the README too large.
https://www.drupal.org/docs/extending-drupal/contributed-modules/contrib...
We could create a documentation page for additional Crawler Rate Limit documentation, to supplement the README.
ORIGINAL post:
I just got DDOS'ed by crawlers, discovered by using one of the great command line log examples from the README:
# Print total, served and rate-limited number of requests per hour on a given date, and percentage blocked
$ for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { percent = NR ? (limited/NR)*100 : 0; printf(" %6d %6d %6d %6d%%\n", NR, served, limited, percent) }'; done
00 8483 7146 1337 15%
01 24585 18290 6295 25%
02 76944 31055 45889 59%
03 131538 32164 99374 75%
[...]
The visits were spread over a huge sub-range of IP's, so it wasn't clear which IP's to block in .htaccess:
$ cat /var/log/apache2/access.log | awk '$9 == 429 {print $1}' | sort | uniq -c | sort -rn
3812 42.201.192.69
3780 42.201.192.27
3750 183.87.99.50
3700 42.201.192.57
3699 42.201.192.82
3698 183.87.96.78
3696 42.201.192.74
3687 183.87.96.79
3686 42.201.192.87
3677 183.87.97.221
[...]
So I split the output by the "." (dot) character, and limited output to first two parts, to group and find the offending parent IP parts:
$ cat /var/log/apache2/access.log | awk '$9 == 429 {print $1}' | cut -d . -f 1,2 | sort | uniq -c | sort -rn | head
282657 42.201
266503 183.87
16423 14.191
3798 40.77
3372 52.167
2484 14.231
2254 113.173
2244 123.24
2141 123.16
2043 14.186
I could add this in the .htaccess, causing the Load Average to go down from 20 to 1:
RewriteEngine on
RewriteCond %{REMOTE_ADDR} ^42\.201 [OR]
RewriteCond %{REMOTE_ADDR} ^183\.87
RewriteRule ^ - [F]
Show commandsStart within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes, plain diff MR !26
Comments
Comment #2
ressaComment #4
ressaComment #5
vaish commentedThanks, @ressa. I feel that we have enough examples in the README to serve as a starting point for building queries that suit user's particular needs. Just like you are doing. Instead of adding more examples to README, I think it would be better to create documentation page for this purpose where more examples could be added without making the README to large. Let me know what you think.
Comment #6
ressaA new doc page under https://www.drupal.org/docs/contributed-modules/crawler-rate-limit is a great idea @vaish, and I created a Guide, and a Page under it.
https://www.drupal.org/docs/extending-drupal/contributed-modules/contrib...
Feel of course free to edit, or add anything, to improve it.
Comment #7
vaish commented@ressa Thanks for creating the documentation page.
One tip. The following oneliner:
cat /var/log/apache2/access.log | awk '$9 == 429 {print $1}' | sort | uniq -c | sort -rncan be written more concisely like this:
awk '$9 == 429 {print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -rnIn general, whenever you see
catas a first command in a pipe, check if the following command can accept file name as an argument.Comment #10
vaish commentedDocumentation link added to the project page.
Comment #11
ressaThanks for the tip @vaish, it's always nice to get the most compact version possible. I really ought to sit down and study awk, it seems like an amazingly versatile tool.
Comment #12
vaish commentedYes, awk is amazing tool. It's worth learning at least the basics. My tip, though, applies in general. For example, another common command where the same applies is grep.
Comment #13
ressaYes, I love grep! I found a great tip on how to only return the text right before or after the instance (
{0,10}= 10 characters). For example, I used it to recently find text strings that needed to be updated for #3610824: Change (disabled) to (uninstalled) on Modules and Themes pages. The file name, line number, and match has different colours in my terminal, so much easier to discern between them than what is shown here: