Problem

We have setup a high-traffic site (and a smaller site, with more to come) for a client, running on Aegir. The infrastructure consists of HaProxy load balancers, Varnish cache servers, and then nginx and PHP-FPM web servers. This runs great, it's lovely to have Aegir integration with nginx. There's one issue we face with this configuration directive Aegir outputs:

limit_conn gulag 32; # like mod_evasive - this allows max 32 simultaneous connections from one IP address

This is useful on setups where the web servers are public facing, but the web servers in this setup receive their traffic from just a few Varnish servers. As soon as real world traffic hits this configuration nginx starts refusing connections from Varnish (due to the configuration directive) and the client's sites go down. To work around this I have been effectively forking Aegir by hacking the template files (provision/http/Provision/Config/Nginx/Inc/vhost_include.tpl.php for example) to increase the conn_limit to 10,000 or some other high number.

Today however, something happened which set the configuration back to the default limit_conn gulag 32 on the web servers (am still investigating what happened/who did it) so the client's sites went down; I am now not the most popular sys admin, since the high-traffic site gets 4,000 visitors a minute and by the time I got the site back up they had lost 120,000 visitors, I'm not privvy to the figures but this is probably a couple of thousand dollars worth of revenue. I mention this not to blame anyone but to show this issue has significant, real-world impact.

Proposed solution

The nginx 'basic' configuration is already in use for the setup mentioned above (which we have found very useful). The limit_conn directive could be omitted in the basic nginx configuration but still included in the default (since it's probably useful for most setups); anyone with the 'basic' nginx configuration who still needs this could write a Drush plugin to inject the limit_conn directive into their configuration.

Thanks!

Comments

Liam McDermott’s picture

Liam McDermott’s picture

Title: nginx behind a Varnish reverse proxy requires forking Aegir » nginx behind a reverse proxy requires forking Aegir
omega8cc’s picture

This is a very old default limit, and we have overlooked this while removing all extra config directives used in the BOA project.

It should be removed altogether, because it causes problems also when your site is behind CloudFlare (so it has been bumped to 555 by default in BOA)

The reason you see this again is that it is set in the vhost template, so basically any task will restore this directive currently.

omega8cc’s picture

Title: nginx behind a reverse proxy requires forking Aegir » Remove too aggressive limit_conn directive in the Nginx config templates
Status: Active » Fixed

Fixed in 6.x-2.x and 7.x-3.x

  • Commit 6f86539 on 6.x-2.x by omega8cc:
    Issue #2259461 by Liam McDermott - Remove too aggressive limit_conn...

  • Commit 0461b1d on 7.x-3.x by omega8cc:
    Issue #2259461 by Liam McDermott - Remove too aggressive limit_conn...
Liam McDermott’s picture

Thanks very much omega8cc, that was some very fast work!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

  • Commit 0461b1d on 7.x-3.x, dev-helmo-3.x by omega8cc:
    Issue #2259461 by Liam McDermott - Remove too aggressive limit_conn...