Make module compatible with Varnish Cache 4.0.x

Varnish 4.0.0 is now out as of April 10. Changes to VCL are here: https://www.varnish-cache.org/docs/trunk/whats-new/upgrading.html

I set up a test server, and installed Varnish 4.0.0 on it.

I set up a test Drupal site, and installed the Varnish module. I changed the port number to the one my varnishd was using as it's terminal connecting port, and got the error "The Varnish control terminal is not responding on port X". I hadn't entered a secret key, so tried that and the same error.

I then uninstalled Varnish 4, and installed Varnish 3. I used the same port / secret key / storage container - every setting - as I had been using for my test of Varnish 4. The module configuration page confirmed "Varnish running". This worked whether I had a correct value, a wrong value, or no value in the secret key field.

So it seems that Varnish 4 has changed whatever handshake the Drupal module is looking for to confirm it can connect to the Varnish terminal. It seems this is the first time a new release of Varnish has changed at this point - when running Varnish 3 on the server, I was able to tell the module to use Varnish 2.0, 2.1, or 3.0 - and it still worked. So if the Drupal module is expecting Varnish 2, but the server has Varnish 3, everything still basically works. But if the Drupal module is expecting Varnish 3, but the server has Varnish 4, the module cannot connect to the Varnish terminal.

{Making issue title clearer}

The next version (1.2.0) of the Varnish PECL package will support Varnish 4. No idea when it will be out, but it could be an efficient way to support multiple versions.

Documentation here.

Interesting. I imagine the module could start using Varnish PECL, but it would be a complete rewrite of most of the module. The way it's done, with sockets, is cleaner, and has the advantage of working without adding an extra server requirement.

The module already has switch statements to handle the occasions when Varnish 2.0, 2.1 and 3 differ. Looking at the code, I can't see any obvious reason why this wouldn't work with Varnish 4.0. I think the "ban" command syntax may have changed slightly, but the code that checks if Varnish is running doesn't get that far; something's failing further back.

I haven't got a test machine to debug this at the moment, but I suspect the material differences are small, as far as this module is concerned. A quicker solution would be for someone to fire up a server that has Varnish 4 on, and then debug the module to see why the status is reporting a failure.

I've implemented Varnish 3 stats using the PECL module: #1195678: Varnish 3 and statistics

It was a trivial feature to add.

Aside from supporting multiple versions of Varnish, the PECL package supports both shared memory and terminal control of Varnish. It's also likely to be slightly faster, not being written in PHP.

That said, supporting the PECL as an option would add some complexity to the module, but not a total rewrite. Simple detection and handing off commands to the appropriate function (terminal or PECL).

I've never used the PECL - can it send ban / purge commands through to the Varnish terminal? That's the main purpose of this module.

Check the link to the documentation in #4. The overview page has a list of all the methods.

I think it is best to leave this module as it is now. The only think that is needed is to add support to Varnish 4.
Also I think this is very important upgrade. Joomla and Wordpress are currently support Varnish 4.

In this new release we have some new interesting long waited options. One them is

Background (re)fetch of expired objects. On a cache miss where a stale copy is available, serve the client the stale copy while fetching an updated copy from the backend in the background.

I think this is one of main option if you want to run high load sites(news, portals, blogs etc...)

Varnish 4.x was released 3 month ago and you still don't have support for it. I think it is big minus for Drupal community expectantly varnish-cache.org company which site is also made by Drupal.

Any news regarding this? Do any maintener working on this?

Subscribing. I hope a support for varnish 4 will be soon available. Thanks a lot to the devs for their hard work.

blackandcode: maybe we can team up to provide the required patches for Varnish 4
I'm looking into it at the moment.

#11 +1, sub and so on :)

Looking and putting up a development environment to try and resolve this issue. Anybody else made any progress or head way?

Subscribing.

Happy coding people!

I'm following this closely and appreciate the work everyone is putting in. I have not yet successfully implemented any version of varnish yet, so I'm a noob and probably wont be too much help, but I am planning on trying varnish soon so I'll be watching the progress here to determine if I use varnish 3 or 4.

Here are the changes I made to get this module to work with Varnish 4.

VCL Changes (I'm only including lines that need to be added to make it work)

sub vcl_deliver {
    #remove headers set for banning purposes.  They are only necessary when stored in the cache.  The end user doesn't need them
    unset resp.http.x-host;
    unset resp.http.x-url;
}
sub vcl_backend_response {
    #set variables for use with banning later on
    set beresp.http.x-host = bereq.http.host;
    set beresp.http.x-url = bereq.url;
}

With these changes, everything works when you set the Module to Varnish version 3.x. The only change required is adding the 4.x option at this point although the 3.x version works just fine. I've attached a patch for this small change.

I also had to apply this patch (https://www.drupal.org/node/2340829#comment-9212271) to get everything working. I'm not sure if this is limited to Varnish 4, but the Drupal $base_path is part of the URL that Varnish stores. If it's removed, nothing will be matched correctly and nothing will be cleared from the cache.

I'm currently using the Ban Lurker option with the Expire Module (Selective Expiration) if that helps anyone.

+1 to the documentation. I would help with testing now if I had a simple step-by-step guide how to set it up.

The '4' => '4.x' will continue to work for 4.0.1 and beyond. The only reason we would have to separate it out is if a 4.2 or 4.3 changed the API and the PURGE/BAN commands had to be handled differently. We could have also changed the '3' => '3.x' to '3' => '3.x+' as the same code works for 3.x and 4.x.

I'm still working through different Varnish scenarios right now. Once I have them all figured out and working in production, I'll circle back with some documentation here.

+1 on RTBC #20
Varnish vcl needs some tweaking but this issue is a minor addition and crucial to get anywhere at all.

After some further tweaking I've discovered that varnish 4 doesn't work without configuring a secret key. I usually don't authenticate over localhost. But the varnish module seems to require me to do so.
Furthermore, here's a varnish 4 vcl to get you started. https://github.com/mattiasgeniar/varnish-4.0-configuration-templates

I am in the process of setting up a new server and would like to use Varnish 4. What is the ETA of this being released?

It looks like version 3 support is going to end in a few months. April 2015.

https://www.varnish-cache.org/content/varnish-cache-306

(Marking Priority -> Major because of the impending EOL for Varnish 3 referred to in comment #27 above).

I've just set up a test server, and installed Varnish 4. My daemon options for Varnish have "-T localhost:6082" in, so the control port is indeed 6082 on localhost. I can confirm that it works, because I can log into the server and type "telnet 127.0.0.1 6082" and I'm accessing the Varnish control process. I'm then told that authentication is required, but that's enough to tell me that the port is working.

I then installed a brand new Drupal site, and made no changes to a vanilla installation. I installed this module, and applied the patch from #20. I went to Home » Administration » Configuration » Development » Varnish, chose Version 4 from the dropdown, entered my secret key (from /etc/varnish/secret), and saved the configuration.

Now, when I load that Varnish admin screen, I still get a status message at the bottom of the screen: "The Varnish control terminal is not responding at 127.0.0.1 on port 6082."

If I uninstall Varnish 4, and reinstall Varnish 3, the Status message at the bottom reads "Varnish running." All that's different between these two scenarios is whether the server is running Varnish 3 or Varnish 4.

This is exactly the same as I reported in comment #2 of this issue. Something more is needed than just adding a new value to the list of Varnish versions.

The reason is that Varnish authentication has changed somehow. I'm getting "Authentication to server failed!" in the dblog when I run the Varnish 4 arrangement.

Apologies for the noise in this thread.

I've solved this - but it was a problem my end, so there's no need to change anything in the module.

That said, it is a difference in behaviour between Varnish 3 and Varnish 4, so it is worth mentioning here. On my test server, my Daemon Opts did not have a /S switch in - which is where the file containing the secret key is specified.

Varnish 3 will take that to mean that authentication is not required. So Drupal will always authenticate.
Varnish 4 will take the absence of that switch to mean that authentication will use some mystery secret key - so Drupal will always fail to authenticate.

Put a /S switch back in, pointing to the secret file, and it all works.

It even works without the patch in #20. As has already been set, the Varnish version in Drupal can be left as "3".

So I did have a problem because I updated Varnish to version 4, but it was my sysconfig file that needed altering, not the Drupal module.

@Lostandfound: Just to clarify, I did need the patch at #2371907: The Varnish control terminal is not responding at theme_hook_original on port to get Varnish 3 or 4 to work with Drupal 7.33(34) - I just didn't mention it in this thread.

With these issues at RTBC, it would be good to have some of them committed so we can test a dev release that rolls up all these patches. (I see nothing has been committed for this module for 9 months or so now.)

I second JamesOakley's previous remark. Would be nice to get this in before Varnish 3 EOL. Especially because some of this stuff is merely cosmetic.
Seeing as Varnish 4 requires the secret. Perhaps we should make some ajax form behaviour that sets the required property on the secret form element if Varnish 4 is selected.

I've added a patch for the ajax behavior. It includes the version number patch and some code consistency fixes.

Forgot to change the status.

Reviewed.

The 4.x option in the drop-down is there.
The Control Key is appropriately optional iff version=="4.x"
The Ajax callback works as designed.

-> RTBC

(One thought - you've made a few other changes to bring the admin.inc file into line with Drupal coding standards. Those changes may be best put into another patch in another issue, so that this patch only changes the necessary lines to solve this issue - but that's my personal preferred working-style, hence not a reason to mark as RTBC).

In this case, there aren't too many coding standards fixes. It's probably not worth the trouble to split these out. I'm happy leaving them in here, and keeping this RTBCed. If there were a lot of these changes, then yes, I think it would be a good idea.

My reasoning for fixing coding standards stuff is simple: while I'm in the code, why not fix it? So long as it's trivial and minor, that is. If it's a big change reviewers have a harder time scrolling through all the changed lines. On the other hand, scanning through coder test results becomes a whole lot easier without all the style warnings. And anyone using an IDE with code sniffer has a better experience too. And I never really see any issues dedicated to coding standards.

But I'll be glad to submit a slim patch for this issue, if maintainers prefer this.

RTBC, both patches work

*sigh* I just can't get it to work. I'm sure that it is nothing wrong with the patch, but as soon as I switch apache to listen on port 8080 and start varnish (set to listen on port 80) the server refuses the connection and serves up the cpanel template for nonexistent pages. On the admin/config/development/varnish page (that I can still access because I have it on https which is of course a different port) it tells me that varnish is running, but it also tells me "The Varnish control terminal is not responding at theme_hook_original on port ." I've applied the patches from these two issues just in case they were causing a problem:
#2371907: The Varnish control terminal is not responding at theme_hook_original on port
#2340829: Leading slash missing from ban URLs
My guess is that this is a varnish issue, but I'm using @basvredeling template from https://github.com/mattiasgeniar/varnish-4.0-configuration-templates with no changes other than changing the port to 8080. My varnish config file is using this for the DAEMON_OPTS:

DAEMON_OPTS="-a :80 \
             -T 127.0.0.1:6082 \
             -f /etc/varnish/default.vcl \
             -t 120 \
             -p thread_pool_min=50 \
             -p thread_pool_max=1000 \
             -p thread_pool_timeout=120 \
             -u varnish -g varnish \
             -S /etc/varnish/secret \
             -s malloc,256m"

Is there anybody that can help? I would be happy to pay for someone to walk me through this and debug.

@deggersten - Does it work with Varnish 3, but not with Varnish 4?

If it does, then it's related to this issue (how to make sure the Varnish module is also compatible with the newer version: Varnish 4).

If it doesn't even work with 3, then it's not related to this issue, and I suggest you open a new support-category issue in the issue queue, and outline your problems there.

Yes, I understand that my issue is probably even out of the scope of this module even so I probably just wont post anything else here. If there is an easy solution I thought it might be nice for others to know here, plus I wanted to see if there is anyone here that can help. Nothing more needs to be said here about this. PM me if you are able to help.

We used Varnish 3.0 several years for high traffic site. After 10 days I succeed to make new Varnish 4.0 and Drupal to work without any module.

As you know new varnish has been rewritten completely and it has new features like background refresh. I used this new option to make it work with durpal without any modules.

I will make a post blog in the next a few days and describe in details what has to be done.

@blackandcode I look forward to your post!

If a node is updated, surely the Drupal site needs to notify the Varnish installation of this?

The only alternative would be for Varnish to poll for a change on every request, but that defeats the benefit of the caching system.

So I'm intrigued to see how you could integrate a Drupal installation to Varnish without some kind of module to handle that feedback system.

Added the patch to latest dev, I will try to get a new release out in some days for now, there some other RTBC issues to also to care of before that.

Hi All, I just got to a stage when I installed varnish for the first time. Everything seems to be working fine, but when I installed this module I realized that 4.0 is not supported. I got to this issue board for this reason. After reading it through I was glad to see that the dev version of the module was just updated to support 4.0. I installed the module, but I get a white screen of death after enabling it. Is it only me? Did I miss something?

Also I am not sure if this module is necessary at all... Varnish seem to be working properly without it. My only "fear" is that the expire module might not work without this module being enabled. (This might not even be a case, I haven't got to that point yet)

Any ideas or thoughts on this? Do I really need this module? ...and if so, why?

Thank you

@Lostandfound Thank you for your answer. Seems like I will have to look around a bit more to figure this out... When I disable the module, the site works as expected. Strange. Anyway, thanks again.

@Dajka: See Blank pages or "white screen of death" (WSOD).

Hi,
I created default.vcl for Varnish 4.x and Drupal 7.x on github.

Please check it.
https://github.com/iraneagle/Varnish-Drupal

Please update this line on the main page of module

We currently support Varnish 2.0, 2.1 and 3.