Problem/Motivation
The X-Drupal-Cache-Tags header's value can exceed the limit of the various components of the stack (web server, reverse proxy). The components in the stack can choose to omit the header (thus causing cache invalidation problems) or choke on it (thus causing WSODs).
So far, this is only happening for sites that are either legitimately very complex (think news sites, which show dense/advanced/computed content that depends on many things) or sites with enormous amounts of content on a single page (think brochureware sites — the cache tags header size in that case is in fact a nice indicator of how slow any browser will render such a page).
Proposed resolution
From #2241377-54: [meta] Profile/rationalise cache tags:
I talked to Wim about this - from Acquia Cloud's perspective it would be fine if the default limit was "no limit" as long as it was easy to change. We could then set the limit in the Acquia hosting include on behalf of our customers to match what our platform supports.
Introduce such a header value size limit; hosting environments can then let Drupal know about their limits. Drupal can then automatically replace the cache tags in the X-Drupal-Cache-Tags header with hashed equivalents. This results in false positives (i.e. invalidating some responses that don't need to be invalidated), but allows the site on this limited environment (or ridiculously complex site)
Remaining tasks
TBD
User interface changes
None.
API changes
None.
Data model changes
None.
| Comment | File | Size | Author |
|---|---|---|---|
| #18 | reroll.patch | 9.28 KB | borisson_ |
| #14 | header_size_limit-2542868-14.patch | 9.28 KB | sorressean |
Comments
Comment #1
wim leersThis should do it.
One unresolved problem in this patch:
Comment #2
wim leersComment #4
wim leersComment #5
dawehnerCan we put the default value into core.services.yml as well?
Comment #6
wim leersWhy? It doesn't need to be — Berdir and I were discussing that at the Londons sprint too: those container parameters in
core.services.ymldon't belong there; they belong only inservices.yml(which is originally a copy ofdefault.services.ymlof course).Comment #7
dawehnerjust add a static var to detect the recursive calls?
Comment #8
fabianx commented#6: That is a good reasoning, but the problem is the upgrade path again:
Think of it like:
variable_get('foo_variable', 'default');
And then in settings.php a:
#$settings['foo_variable'] = 'some-value';
Container parameters don't have a default, so core.services.yml _is_ our default as container parameters are overwritten by whoever comes last in $settings['container_yamls'].
If we don't have a default in core.services.yml or for a core module in module/module.services.yml then updating core versions fails as the parameter is missing.
That is why we need both IMHO.
Comment #9
wim leers#7: don't we want to avoid adding state to services? But yeah, I don't have other ideas either.
Comment #10
paulmckibbenRe #7: I tried the static variable approach, and it seems to be working.
Comment #11
sorressean commentedJust uploading this to see if it fails tests.
Comment #12
sorressean commentedGo testbot go! Make me proud!
Comment #14
sorressean commentedHere's another attempt at this.
I put the default value in core.services.yml. I'm also thinking there should be a test for this, but I'm not really sure how to make that work, so any thoughts would be great.
Thanks,
Comment #15
sorressean commentedComment #16
wim leersCreate a test route whose controller does something like this:
That then allows you to easily generate responses with an
X-Drupal-Cache-Tagsheader of various lengths, and in the test you can change the value of the limit to verify that it handles it correctly.Comment #18
borisson_Reroll of #14 attached.
Comment #21
sunhttp.response.header.max_lengthwould be a much more appropriate parameter name.Comment #22
wim leersThe biggest part of this problem is being addressed at #2527126: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them.
Comment #23
wim leers#2527126: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them was just fixed.
The solution we arrived at there is to never send the
X-Drupal-Cache-Tagsheader except for debugging (and tests; because otherwise we would break existing tests).Since this then exists for debugging purposes only, it is pointless to then limit the size by hashing, because that means you can no longer see the cache tags, which defeats the purpose of showing them for debugging.
Therefore marking "works as designed".
Comment #24
berdirUhm. And actually using them for varnish etc. cache invalidation? :)
I'm not sure if we need this or not, but that argument doesn't really work ;)
Comment #25
moshe weitzman commented@Berdir - the idea is that cache tag headers will be emitted by Cloudflare module, Varnish module, etc. Core is no longer in the business of sending production headers.
Comment #26
wim leersWhat Moshe said. Precisely because each of those services have their own expectations regarding header name (Fastly:
Surrogate-Keys, CloudFlare:Cache-Tag…) and header value format (space separated, comma-separated …).Exactly that is mentioned in the CR too: https://www.drupal.org/node/2592471, point 2.
Comment #27
wim leersIn other words: even if we implement a generic setting, it'd still be up to those modules to respect it.
So… I guess we could do it, but it'd be a relatively strange API/setting/container parameter at that point: it would be mostly for things *outside* core.