Problem/Motivation
I wanted to add a percent column, and ended up with this, after many experiments:
for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "/2026:13:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END {limited != 0 ? percent = ((limited/NR)*100) : percent = 0; printf("%6d %6d %6d %6d%\n", NR, served, limited, percent) }'; done
... so this is what I added:
; limited != 0 ? percent = ((limited/NR)*100) : percent = 0
... and
%6d%
... and
, percent. The result:
00 188 108 80 42%
01 289 128 161 55%
02 190 150 40 21%
03 163 149 14 8%
04 179 141 38 21%
05 187 149 38 20%
06 212 117 95 44%
07 171 101 70 40%
08 146 67 79 54%
09 148 70 78 52%
10 139 58 81 58%
11 126 57 69 54%
12 134 118 16 11%
13 139 106 33 23%
14 159 91 68 42%
15 140 100 40 28%
16 117 70 47 40%
17 157 81 76 48%
18 131 53 78 59%
19 154 59 95 61%
20 209 120 89 42%
21 221 117 104 47%
22 235 193 42 17%
23 176 119 57 32%
24 237 167 70 29%
25 236 87 149 63%
26 165 79 86 52%
27 187 98 89 47%
28 277 161 116 41%
29 187 91 96 51%
30 222 128 94 42%
31 214 67 147 68%
32 173 112 61 35%
33 0 0 0 0%
34 0 0 0 0%
35 0 0 0 0%
36 0 0 0 0%
37 0 0 0 0%
[...]It could get added to the last example like this:
# Print total, served, number and percent of rate-limited requests for each
# minute in a given hour.
# This example uses 17th hour (5 pm) on 15/Jun/2025. Period from 17:00 to 17:59.
# Columns:
# - Hour in a day (in 24-hour format)
# - Total number of requests
# - Number of allowed/served requests
# - Number of rate-limited requests.
# - Percent of rate-limited requests.
for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "15/Jun/2025:17:$i:" /var/log/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { limited != 0 ? percent = ((limited/NR)*100) : percent = 0; printf(" %6d %6d %6d %6d%\n", NR, served, limited, percent) }'; doneSteps to reproduce
Proposed resolution
Remaining tasks
User interface changes
API changes
Data model changes
Issue fork crawler_rate_limit-3590089
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #3
ressaComment #6
vaish commentedMerged. Thank you @ressa. Note that I had to make some changes to your code. I suggest you use the final (merged) version of that bash one-liner.
Comment #8
ressaIt looks great and works well, thank you @vaish.