Hello,
I'm trying to debug a server and I was hoping you could provide some guidance on how I could debug the following Authcache error:
Everytime anyone makes a request I get the following error in my drupal logs, and all of my Varnish requests MISS:
access denied 20 Jun 2014 - 2:57am authcache-varnish-get-key Anonymous (not verified)
Do you know what problem this could be signalling? My understanding is that Varnish is trying and failing to pull up the Authcache key to get the correct page?
I've been looking into varnishstat and it's registering cache hits correctly (i.e. repeated views of a page, creates a cache hit) but I'm still getting the cache miss in headers.
Frustratingly this problem only exists on my live server. My dev server works fine.
My debugging so far
I'm running a LEMP server with
Varnish 3.05
Memcached
APC
PHP-FPM
Nginx
Authcache 7.x-2.0-beta3 (Modules: block, comment, field, form, menu, varnish storage backend, views, voting api).
As I mentioned the problem only exists on the live server, and so I think I've managed to rule out Varnish and my drupal install as the problems:
The default.vcl config files are identical.
I migrated across a copy of my live drupal installation to my dev server and it still works.
So the problem must be some other link in the chain, but I'm unsure what exactly the error is telling me and so where to look next.
Comments
Comment #1
splitsplitsplit commentedComment #2
znerol commentedAccess to the
authcache-varnish-get-keyis controlled by the function authcache_varnish_request_validate(). This function checks whether the request came in through the reverse proxy. Therefore please check the following things:settings.phpfile? Especially$conf['reverse_proxy']and $conf['reverse_proxy_addresses'])Comment #3
splitsplitsplit commentedOk so I figured out what it is but I'm not sure how to fix it. I have the following block in my Nginx config
It sets the IP correctly of all the requests coming through Varnish so I can still use Fail2ban to ban people. (Otherwise all my nginx logs are from 127.0.0.1 because they're run through Varnish).
Presumably because it's passing through the correct IP the settings PHP chunk:
Is then blocking them because they're not now from 127.0.0.1. Is there some way to tell Drupal that these requests are coming from Varnish?
Comment #4
znerol commentedIn this case you can suppress validation of the reverse proxy by adding the following line to
settings.php:Then you also may remove
$conf['reverse_proxy']and$conf['reverse_proxy_addresses'].Comment #5
znerol commentedAdded a notes section on the handbook page covering thenginx HTTP realip module. Feel free to improve it if necessary.
Comment #6
splitsplitsplit commentedYep seems good, I added an explanation of how to check if you have real IP enabled in case someone isn't 100% sure.
Is there any particular trade off I should be aware of by turning off that validation?
Comment #7
znerol commentedYou must make sure that your users never bypass varnish. For example when introducing SSL that might be relevant.
Comment #8
splitsplitsplit commentedHmm yes that is a problem, SSL traffic comes in through 443 and bypasses Varnish completely. Presumably that means they can grab from the cache and everyone is trusted.
Gonna have to come back to this, I guess the ideal solution would be for Nginx to pass requests through to Drupal as 127.0.0.1 and only use the real IP header for logging, but I need learn how to do it or even if it's possible.
EDIT: This might be the solution, to just change the log file reporting, will report back when I get it working:
http://linuxaria.com/article/how-to-log-the-correct-ip-having-varnish-an...
Comment #9
znerol commentedYou also could disable the reverse proxy validation only if people are not coming in via HTTPS:
Or, you may implement your own validation and then use it like this in
settings.php:Comment #10
splitsplitsplit commentedOk so I got the log solution working. It's super simple, borrow the post I listed above.
In your http block in nginx you declare a new log format:
Then in your virtual host file in
/etc/nginx/sites-available/you add the following option to your log settings:access_log /var/log/nginx/access.log varnish_log;Compromises: You can't change the log format of the error log, which means if you're using basic authentication as an extra layer of protection for your backend you can't use fail2ban to monitor it.
I think I like your method slightly better though, because it avoids that compromise completely.
Presumably in that case my settings.php file would look like:
Comment #11
znerol commentedIf users accessing the site via HTTPS bypass varnish completely, then you also do not need the
$conf['reverse_proxy']and$conf['reverse_proxy_addresses']directives.Comment #12
splitsplitsplit commentedOh of course, no need for the variable if you've disabled the validation.
I've updated this in the handbook.
I left out the log solution because it fixes the same problem that real IP fixes but without requiring any changes to Authcache, I thought it might be a little off topic.
Thanks very much for helping me sort all of that znerol, that's been really helpful.
Comment #13
znerol commentedThanks for helping with the documentation.
Comment #14
znerol commentedI'm trying to improve the situation for Nginx users, would appreciate feedback on #2354205: Provide an alternative way to verify whether requests came in through Varnish.