Hello Vaish,

I've been using CRL for a fleet of Drupal sites to great effect.

One problem we were running into was rate limiting being applied to our team of editors who were all working from the same IP address, using browsers with identical user agents.

When enough editors were working on the site at the same time, CRL limited them, which wasn't ideal.

The attached patch allows an administrator to specify IP addresses (either as individual addresses, or a range) which should never be rate limited, as a comma-separated list - e.g.:

1.2.3.4,1.2.3.0/8,5.4.3.2

The patch also updates the Readme accordingly.

Hopefully this will be useful to someone!

/Alex

CommentFileSizeAuthor
#8 3409793-white-list-ip-addresses-8.patch5.42 KBroman.haluska
#5 3409793-white-list-ip-addresses-5.patch5.43 KBmgaskey
#3 DD-1698-crawler_rate_limit-Allow-administrators-to-specify-IP-addresses-and-ranges-which-should-never-be-rate-limited.patch5.9 KBalexgreyhead
#2 DD-1698-crawler_rate_limit-Allow-administrators-to-specify-IP-addresses-and-ranges-which-should-never-be-rate-limited.patch.txt5.9 KBalexgreyhead

Issue fork crawler_rate_limit-3409793

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

alexharries created an issue. See original summary.

alexgreyhead’s picture

alexgreyhead commented
StatusFileSize
new DD-1698-crawler_rate_limit-Allow-administrators-to-specify-IP-addresses-and-ranges-which-should-never-be-rate-limited.patch.txt5.9 KB
alexgreyhead’s picture

alexgreyhead commented
StatusFileSize
new DD-1698-crawler_rate_limit-Allow-administrators-to-specify-IP-addresses-and-ranges-which-should-never-be-rate-limited.patch5.9 KB
vaish’s picture

vaish commented

Hello Alex,

I'm really glad to hear you are finding the module useful. Thanks for the patch. I still didn't have time to review it but this feature is something that I find useful addition to the module. Would you mind creating a merge request. It might be easier for me to review it that way.

Best,
Vaish

mgaskey’s picture

mgaskey commented
StatusFileSize
new 3409793-white-list-ip-addresses-5.patch5.43 KB

Re-rolling patch in #3 for latest updates in beta2 release.

mgaskey opened merge request !5

mgaskey’s picture

mgaskey commented

Hi Vaish

I've added an MR with Alex's changes, and my re-roll for the latest version.

roman.haluska’s picture

roman.haluska commented
Version: 3.0.0-beta1 » 3.0.0-beta2
StatusFileSize
new 3409793-white-list-ip-addresses-8.patch5.42 KB

Small doc fix

vaish’s picture

vaish commented
Title: [Patch provided] Allow administrators to specify IP addresses and ranges which should never be rate limited » Allow administrators to specify IP addresses and ranges which should never be rate limited

  • vaish committed 32f56e64 on 3.x authored by mgaskey 
    Issue #3409793 by mgaskey, alexgreyhead, vaish: Allow administrators to...
vaish’s picture

vaish commented

Thanks everyone. Please note that merged version contains some important changes compared to the initial MR.

1. Allowlist setting has been changed to an array and renamed:

    $settings['crawler_rate_limit.settings']['ip_address_allowlist'] = [];

2. Function updateClientIpXForwardedFor() has been removed. I believe that this functionality does not belong here. Figuring out real IP address should be handled for the whole Drupal install and not for a single module. Additionally, this particular implementation has been focused on Cloudflare and may or may not work with other proxies. Note that Symfony's getClientIp() method can read the client IP address from the "X-Forwarded-For" header when trusted proxies were set via "setTrustedProxies()". Drupal exposes this as a setting in the settings.php file. See:

vaish’s picture

vaish commented
Status: Needs review » Fixed

Thanks for the patch, @roman.haluska. Note that your patch came in at the same time I made some changes and removed the function you patched from the final version. See my previous comment for more details.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.