Add authcache key cookie so that it can be used by upstreams proxies

Well, not exactly, at least I think so. In the way this patch adds it, which is the same of what you do in

authcache_authcache_account_exclude().

If authcache is busted, it means that no "caching is being done", so the no-cache headers are also being set, so in those cases, it's fine not to have anything to vary on, because you are also sending the no-cache header, so there's nothing to attempt to vary on.

Or am I understanding this in the wrong way?

I tried it on a setup, So I am getting and varying on x-authcache-roles, on anon with sessions. The minute I log in with a different role in which no x-authcache-roles is set, I am still getting the proper logged in page. I think because an empty Vary it's still a valid Vary combination.

I edited the previous comment with more info, sorry i didn't add another comment, just in cased you missed it. Thanks for getting back so quickly (before I got to submit the patch:).

I am pretty sure it's the other way around.

That's why the vary is set server side by Drupal and your module.

The "Vary" header field in a response describes what parts of a
request message, aside from the method, Host header field, and
request target, might influence the origin

See
- https://tools.ietf.org/html/rfc7231#section-7.1.4
- https://www.fastly.com/blog/best-practices-for-using-the-vary-header
- https://varvy.com/mobile/vary-user-agent.html

you are so very right. I got this wrong. Thanks for pushing the explanation. It should be a cookie with the authcache roles then with some transformation in the vcl t convert that into a custom request header. I will see what the varnish module does but I believe at least the cookie should be present regardless of the backend module cache.

Thanks! I just looked properly on the varnish module, and you are right that they are doing something similar, however, the logic is complex and really applies to something more out of the box, but it's just too custom for implementing it on mostly any cdn as reverse proxy as they normally have their own things and limitations.

I think fastly does support ESI but I am not very interested at the moment as we are sorting the dynamic bits of the page on our own, so I don't really need it. What I would like to have, is something general and relatively simple that I could use with fastly (and really any other reverse proxy for that matter).

The authcache_varnish module generates a callback to get the key, it's a bit too troublesome, as the key, imho, could be simply sent on a cookie and then transformed on the cache side.

I noticed a lot of checks that happen on

authcache_varnish_request_validate().

Are those really necessary, is showing the cache key to the outside world any security flaw? It doesn't look like that at a first thought.

So doing something in the like of:

  if (authcache_account_allows_caching()) {
    // Set the key on a cookie so that it can be used by cache proxies transformations
    setcookie('Drupal.authcache.key', authcache_key(), authcache_key_lifetime());
  }

  // Add Vary header. (this is really optional, it could potentially not be added on the module as it can be configured with proper $conf[] variables, or even entirely on the cache side.
  // drupal_add_http_header('Vary', 'X-Authcache-Key');

I think having that for regardless of the backend (as in the main authcache module) would help.

I can do this on a custom module of my own I just think it's a nice addition to authcache, if you agree and unless you have a strong security concern on leaking the authcache key to the outside world, I can submit a patch with this approach.

Only having this information in the cookie, allows to do what fastly explains on the Vary: Cookie section.

Basically it's:

1. transforming the cookie value into a custom http request header
2. adding the custom header to the vary so that the cache use it
3. remove the vary header before sending the response back to the browser.

There in fastly there are other bits that's done although I wouldn't do all of that.

The above vcl is a very short addendum to fastly and probably any other varnish configuration, so it gets really simple unless I am missing something else out.

I think I am starting to understand what I really want to do and how/where to do it and there was a bit of confusion on this issue as how I understood the design of authcache, still definitely not following everything but getting there. Caching is definitely a complex subject.

The main thing I notice is that i wasn't clear in what I was really going for. I don't want to create a new backend for fastly, because it's really not needed. I underestand now that what the varnish module does is leaving the storing of the cache pages entirely to varnish and that's why it has that many logic.

However what I like is to still use the builtin cache, which works very good, but also use fastly for its cdn and a second caching layer.

They could even have different configuration as far as ttl goes and all.

The only thing I see needing for this is to have the builtin cache expose the key in a cookie so that it can be used by fastly. And eventually any other varnish upstream or any other really. I wonder if there might be extra things needed for this to work properly, but I am not quite sure.

I thought this was meant to be globally, like my attempt with X-Authcache-Roles before, but I now see it has to be done within the builtin cache.

I do wonder where is the proper place to add such a cookie. I mean simply adding a cookie that holds the key.

Would adding it in

authcache_builtin_authcache_backend_key_set()

be enough, or that's only for when the key changes?

If so, where else do I need to add it and is there a proper way of adding such a cookie?

I see boost use autcache_add_cookie, is that what I should be using?

On second thoughts, doing an simplish authcache_fastly, or even and authcache_cookie or authcache_generic could be simple and not a bad idea, although adding it to builtin could be harmless/useful and help me in building this other posible module.

Have a look at https://www.drupal.org/sandbox/hanoii/2786375 if you can.

Basically is mostly we discussed here. A dumb backend simply exposing a cookie and a general recipe that can be replicated on serveral upstream reverse proxies.

I still wonder if something so simple shouldn't be on authcache core.

Like adding the cookie regardless of the backend, and maybe having a blank or dummy backend you can enable.

I will see if I come up with some useful authcache_debug info addition, although not sure if possible.

Also why do you check for the cookie when you actually want to use a custom request header?

The custom request header is nearly transparent, and I wanted to make it so that it works the same regardless of that. By using cookie it also allows me to debug the module better than the header. Was planning on making it httponly, although I thought maybe the key is eventually useful to be accesible through JS?

I feel that the current content of the sandbox tries to be two things: A simple generic backend for reverse proxies (which must be carefully configured though) and at the same time an example on how to integrate fastly.

It is two things, but to be honest, is really the former, a generic one.

With this, if I were to set up a local varnish, I wouldn't use the varnish backend, I would use this module and add the vcl property.

All of the fastly references is because I am using fastly for it. And I am also using the fastly module, but it's not a fastly integration. I do provide a fastly vcl example, but even that is tricky, because it depends on a fastly vcl template with specific tags. The minute they change anything on those tags, the boilerplate my change.[1]

With the ever growing list of CDNs and its different caching techniques, something like this makes it really standard.

I cannot comment on the VCL, I'm not familiar with fastly dialect.

Besides the fastly.vcl, I will also add a varnish.vcl, which will be basically the vcl example on the project page. It's not fastly dialect, it's simple varnish dialect. The recipe in the project page works the same for any reverse proxy.

By adding this into core (just setting and maintaining the cookie) you are allow to do this generic setup, and you get one more benefit, which is that you can also do so regardless of the backend you use. So you can set a builtin cache, and also use fastly with the same recipe. In that case you would have two different caches working at the same time. It's odd, but allows you to do nifty things. I'd reconsider. It's just so simple that it's almost a pity not to do it. And adding a cookie is really backend agnostic. You can use it or not, but it's there.

I am still working out a few issues with mixed http/https sessions, and yes, sorting it out with fastly, but the beauty of this cdn is that it's making you respect a standard to vary on the Vary header, so whatever you do, works the same for anything else.

Will soon to submit another patch for the cookies part on authcache.

[1]: https://docs.fastly.com/guides/vcl/mixing-and-matching-fastly-vcl-with-c...

Fair enough.

The Authcache CDN is really Authcache upstream, my (probably poor) choice of name is only because CDN is not really the case. The upstream module can be used with a local varnish, a local nginx, anything really as long as you are allow to modify with headers and Vary.

Will continue shaping out this module and release it as such.

Thanks a lot for the ongoing discussions. As you said is a complex subject :).

@Wim Leers glad and happy you find the issue interesting!!

Updated the issue summary a bit, still working on it but I think it's pretty stable atm.

Just promoted the module to https://www.drupal.org/project/authcache_upstream

And I guess we can close this issue