Problem/Motivation

The bash one liner examples under "Logging of rate limited requests" is yet another example of the exemplary documentation of this amazing module: https://git.drupalcode.org/project/crawler_rate_limit#logging-of-rate-li...

You can pull out and compare any data points, or adjust the examples to get the desired data.

There does seem to be missing a space between the "Hour" and "Total" columns, for the "Print total" example for a given date (I have log-files by date, so this gets today's log):

Now:

# for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { printf("%5d %5d %5d\n", NR, served, limited) }'; done
0015980 14795  1185
0115084 14441   643
0218369 16673  1696
[...]

Use " %5d %5d %5d\n" instead?

# for i in $(seq -f "%02g" 0 23); do printf "%02d" ${i#0}; grep "/2026:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++}} END { printf(" %5d %5d %5d\n", NR, served, limited) }'; done
00 15980 14795  1185
01 15084 14441   643
02 18369 16673  1696
[...]

Could this also be useful for the "each minute in a given hour" example?

Steps to reproduce

Proposed resolution

We could add a space before the Total number column?

Also, we could add this tip?

For dates stored in individual log-files, instead of grep'ping for a date, you might use cat access.log for today, cat access.log.1 for yesterday, and zcat access.log.2.gz for two days ago.

Remaining tasks

User interface changes

API changes

Data model changes

Issue fork crawler_rate_limit-3584920

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

ressa created an issue. See original summary.

ressa opened merge request !18

ressa’s picture

ressa
he/him
commented
Status: Active » Needs review

vaish made their first commit to this issue’s fork.

  • vaish committed 35132340 on 3.x authored by ressa 
    fix: #3584920 Add missing column-separator space in bash one liners

By...
vaish’s picture

vaish commented
Status: Needs review » Fixed

Thanks @ressa. I updated remaining one-liners that had the same issue and increased number of digits per column to accommodate larger numbers. Your tip about log files per date, I converted into set of additional examples in order to make the usage more clear.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

ressa’s picture

ressa
he/him
commented

Thanks @vaish, it looks great!

ressa’s picture

ressa
he/him
commented

I wanted to add a percent column, and ended up with this, after many experiments:

for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "/2026:11:$i:" /var/log/apache2/access.log | awk '{if ($9 == 429) {limited++} else {served++} ; limited != 0 ? percent = ((limited/NR)*100) : percent = 0} END { printf("%5d %5d %5d %5d%\n", NR, served, limited, percent) }'; done

... so this is what I added:

; limited != 0 ? percent = ((limited/NR)*100) : percent = 0
... and
%6d%
... and
, percent. The result:

00   188    108     80     42%
01   289    128    161     55%
02   190    150     40     21%
03   163    149     14      8%
04   179    141     38     21%
05   187    149     38     20%
06   212    117     95     44%
07   171    101     70     40%
08   146     67     79     54%
09   148     70     78     52%
10   139     58     81     58%
11   126     57     69     54%
12   134    118     16     11%
13   139    106     33     23%
14   159     91     68     42%
15   140    100     40     28%
16   117     70     47     40%
17   157     81     76     48%
18   131     53     78     59%
19   154     59     95     61%
20   209    120     89     42%
21   221    117    104     47%
22   235    193     42     17%
23   176    119     57     32%
24   237    167     70     29%
25   236     87    149     63%
26   165     79     86     52%
27   187     98     89     47%
28   277    161    116     41%
29   187     91     96     51%
30   222    128     94     42%
31   214     67    147     68%
32   173    112     61     35%
33     0      0      0      0%
34     0      0      0      0%
35     0      0      0      0%
36     0      0      0      0%
37     0      0      0      0%
[...]

It could get added to the last two extended examples like this:

# Print total, served and rate-limited number of requests for each minute in a
# given hour.
# This example uses 17th hour (5 pm) on 15/Jun/2025. Period from 17:00 to 17:59.
# Columns:
#   - Hour in a day (in 24-hour format)
#   - Total number of requests
#   - Number of allowed/served requests
#   - Number of rate-limited requests.
#   - Percent of rate-limited requests.
for i in $(seq -f "%02g" 0 59); do printf "%02d" ${i#0}; grep "15/Jun/2025:17:$i:" /var/log/access.log | awk '{if ($9 == 429) {limited++} else {served++}; limited != 0 ? percent = ((limited/NR)*100) : percent = 1} END { printf(" %6d %6d %6d %6d\n", NR, served, limited, percent) }'; done

Mentioning the option and show the bits below the two extended is another option, but maybe too cryptic for most users?

vaish’s picture

vaish commented

This issue is already closed. Could you please open the new one? I may consider adding percent column to one of the examples. Btw, your code for calculation of percent should be moved to the END block. Currently, it's located in the block which is being executed for each line of the log file. Because of that percent is being unnecessarily recalculated on each iteration instead of once at the end.

ressa’s picture

ressa
he/him
commented

Thanks for the feedback, that placement makes a lot of sense and fixed a strange behaviour as well, and I created a new issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.