Add support for varnish (VCL and ESI)

does authcache esi work with the drupal.org/project/esi module?
does authcache varnish work with the varnish module?

do I need to set a backend when I use authcache_esi ?
Do I use it instead of authcache_ajax?

so authcache_esi and authcache_varnish or authcache_ajax

@znerol: thank you for adding Varnish support to Authcache. I'm having some trouble following the setup for it though. It's not clear when/how to use portions of the included VCL examples and what to incorporate into my already existing Varnish VCL. If you could provide more detailed setup instructions, it would be very helpful. FWIW, this is our current VCL. Thanks!

@znerol: yes, I see a number of different VCL files and it's unclear what order to add these, etc. More detailed instructions would be great and I could even write a patch to explain this better for others. We have an existing VCL as I mentioned that is used for many sites (vhosts) right now, and have to add Authcache Varnish for a single site on that host. Can you explain how to best incorporate these files into my existing VCL above, or are you saying to combine them and load them conditionally for just this site? Thanks.

@znerol: Ok, I first tried adding the Authcache logic to my existing VCL and then decided to just start with the Authcache Varnish VCL bits to see how it behaved. Here is the VCL I have loaded, and it's a compilation of the VCL files included with the module, except for the ESI bits: https://gist.github.com/vinmassaro/e01d902aa08d62f36c17

I am using the latest Authcache 7.x-2.x. I have enabled these modules:

• Authcache
• Authcache Debug
• Authcache Varnish
• Authcache Personalization API
• Authcache Form
• Authcache Menu
• Authcache Search

I added a cache hit/miss header from my original VCL so I could see if Varnish is returning from cache. When logging in with a role set up with Authcache, it does seem to cache and return from Varnish correctly (based on the X-Varnish-Cache and X-Varnish-Hits headers I set), but the Authcache Debug block always displays Cache Status: "Miss", with no reason listed. I'm getting an X-Varnish-Cache: HIT header and X-Varnish-Cache-Hits value greater than 1, so I know the page is being returned from Varnish. Until I added these headers to display, I didn't think it was working based on the output from the Authcache Debug block. Do you see any misconfiguration or could this be a bug in Authcache Debug?

authcache_debug.js checks for the cache_render cookie:

    // If cache_render cookie is set, we did hit the page cache.
    if ($.cookie("cache_render") && $.cookie("cache_render") != "get") {
      cacheRenderTime = $.cookie("cache_render");
      info['Cache Status'] = 'HIT';

      if (cacheRenderTime == 'hit') {
        alertColor = 'green';
        cacheRenderTime = '?';
      }
      else if (cacheRenderTime < 30) {
        alertColor = 'green';
      }
      else if (cacheRenderTime < 100) {
        alertColor = 'orange';
      }
      else if (cacheRenderTime > 100) {
        alertColor = 'red';
      }
    }

Where/how is this cookie set? Does it need to be accounted for in the VCL?

I thought the VCL was easier to digest once I had all of them together in one file. Here is a patch that combines them into one file and adjusts the README a little bit.

I'm away this weekend but will take a look and give some feedback in a few days. Thanks!

Yes, I think this was easier to adapt as one file. If you are using Varnish already and adding Authcache support, you most likely already have an existing VCL. I think Drupal 7 Varnish 3 VCLs are already available on the internet as good examples of where to start, so it's easier to see the Authcache Varnish logic in one file and integrate the pieces into an existing VCL.

I tested the latest dev release with the updated VCL and the responses are working correctly. I am getting cached responses back from Varnish based on the X-Varnish-Cache and X-Varnish headers. Here is how I am testing:

1. Give two users (User A and User B) the role 'staff'. Enable authcache for the 'staff' role.
2. In Firefox, log in with User A. Visit /test-page.
3. In Safari, log in with User B. Visit /test-page.
4. Examine headers in both browsers. If the second value of the X-Varnish header matches for both users, authcache_varnish is working correctly. I also check that X-Varnish-Hits is greater than 1 and for an X-Varnish-Cache: HIT header. I am setting these extra headers in vcl_deliver based on my existing VCL.

The authcache_debug block still seems to be broken, since I never get a green status and Cache Status: HIT, just yellow and Cache Status: MISS. Based on this from example.vcl:

sub vcl_deliver {
  if (obj.hits > 0 && req.http.Cookie ~ "(^|;)\s*cache_render=get\s*($|;)" && resp.http.X-Generator ~ "Drupal") {
    set resp.http.Set-Cookie = "cache_render=hit; Path=/";
  }
}

It appears that cache_render=get cookie never gets set from authcache_debug.js to set cache_render=hit.

Do you think it makes sense to have the authcache_varnish module set a persistent header that we can detect inside vcl_recv, to wrap some of this logic in? Something like X-Authcache-Varnish: enabled if the authcache_varnish module is enabled?

As I mentioned earlier, I'm trying to integrate this into an existing VCL that sits in front of a few hundred sites on university Drupal infrastructure. This configuration is set up to cache requests for anonymous users. If I want to to add support for authcache that will be used on just 1 site, it doesn't make sense to do this for all sites NOT using authcache_varnish, right?

  if (req.restarts == 0) {
    // Retrieve the authcache-key from /authcache-varnish-get-key before each
    // request. Upon vcl_deliver the authcacke-key is copied over to the
    // X-Authcache-Key request header and the request is restarted.
    unset req.http.X-Authcache-Key;
    set req.http.X-Original-URL = req.url;
    set req.url = "/authcache-varnish-get-key";
    set req.http.X-Authcache-Get-Key = "sent";
  }

It looks like I can also restore caching for anonymous users (and authcache_varnish still works correctly for enabled roles) by commenting out:

sub vcl_fetch {
  // Do not cache a response when X-Authcache-Key is empty or missing on the
  // *request*.
  if (!req.http.X-Authcache-Key && req.http.X-Authcache-Get-Key == "done") {
    set beresp.ttl = 0s;
  }
}

You should not need to comment out anything. Authcache Varnish should work for anonymous traffic too. The callback /authcache-varnish-get-key is supposed to return $base_root when accessed by an anonymous user.

I should clarify: Authcache Varnish works for anonymous traffic with the included VCL IF you enable caching for the anonymous role via the module. I wanted anonymous caching to continue to work on other sites that are NOT using Authcache. All sites use the Varnish module and set $conf['cache'] = 1, so they are set up to cache for anonymous users automatically. I would prefer to not maintain separate VCLs and conditional loading logic. My hope was to add the Authcache VCL bits so that it would work for both cases, whether a site has Authcache enabled or not.

I think the 2nd snippet in #19 would work for sites using Authcache, but would apply to all other sites where a user has logged in and you aren't using Authcache. All of the Varnish 3 VCLs for Drupal that I've seen strip out all but the session cookie and bypass Varnish when logged in.

Any ideas about how to fix authcache_debug for authcache_varnish? I can open a separate issue if you want. Thanks!

I'm going to test this out later today and will follow up with a separate issue if authcache_debug is still not working.

This is an important requirement of a project we're working on, so I appreciate the fast response and help with this!

Thanks, I tested the latest dev release and it seems to be working correctly for me aside for the authcache_debug. I opened a separate issue for that. I'll follow up if I run into anything else. Thanks again!

I just noticed an issue that must have been going on but I had not noticed before. We use CAS to authenticate into our sites and what I am seeing is that when a user with the administrator role logs in at /cas, it is returning back a cached, anonymous user home page. When a role that has authcache enabled logs in, it returns them a cached version correctly for their role. I have authcache enabled for the anonymous role, as well as a few special roles.

I tried adding cas and cas* to the authcache page exclusions but it didn't help. I created a Drupal core user with the administrator role, and when I log in at /user, it works fine and logs me in as an administrator since I can see the admin toolbar, etc. When I test it on a site that doesn't use Authcache at all, logging as an administrator with CAS works fine. In vcl_recv, /cas.*$ is part of the custom pass rules. Any ideas?

FWIW, if I disable anonymous caching in Authcache and I log in to CAS as an admin, it works.

I'm not entirely sure what is causing it, to be honest. I'll open up a separate issue.

I noticed that when I added $conf['reverse_proxy_addresses'], I was getting redirect loops when trying to log in with CAS. I updated to the latest Authcache dev and loaded the Authcache example.vcl and selectively commented portions in vcl_recv until I narrowed down the problem. Commenting out the entire 'BEGIN default.vcl' block fixes the problem and allows me to log into CAS. Should this block be in the middle of vcl_recv?

Ah, that must be my problem then. I have MAMP/Apache running 127.0.0.1:8080, Varnish started at 127.0.0.1:6082 and listening on 127.0.0.1:80. My client IP is obviously 127.0.0.1. I just followed your instructions and think I have this working again. Spent way too long on this tonight, thanks!

I have MAMP/Apache now at 127.0.0.1:8080, Varnish at 10.0.0.1:80 connecting to my backend at 127.0.0.1:8080, and added an additional IP to my wireless device. Also have an entry in /etc/hosts for 10.0.0.1 d7. I keep getting the MAMP start page at 10.0.0.1 as if my vhost isn't working. Does this configuration sound right? Also, wouldn't $conf['reverse_proxy_addresses'] need to contain 10.0.0.1, not 127.0.0.1 since the reverse proxy is now at 10.0.0.1?