Problem/Motivation

The X-Drupal-Cache-Tags header's value can exceed the limit of the various components of the stack (web server, reverse proxy). The components in the stack can choose to omit the header (thus causing cache invalidation problems) or choke on it (thus causing WSODs).

So far, this is only happening for sites that are either legitimately very complex (think news sites, which show dense/advanced/computed content that depends on many things) or sites with enormous amounts of content on a single page (think brochureware sites — the cache tags header size in that case is in fact a nice indicator of how slow any browser will render such a page).

Proposed resolution

From #2241377-54: [meta] Profile/rationalise cache tags:

I talked to Wim about this - from Acquia Cloud's perspective it would be fine if the default limit was "no limit" as long as it was easy to change. We could then set the limit in the Acquia hosting include on behalf of our customers to match what our platform supports.

Introduce such a header value size limit; hosting environments can then let Drupal know about their limits. Drupal can then automatically replace the cache tags in the X-Drupal-Cache-Tags header with hashed equivalents. This results in false positives (i.e. invalidating some responses that don't need to be invalidated), but allows the site on this limited environment (or ridiculously complex site)

Remaining tasks

TBD

User interface changes

None.

API changes

None.

Data model changes

None.

Comments

wim leers’s picture

Status: Active » Needs review
Issue tags: +Needs tests
StatusFileSize
new9.75 KB

This should do it.

One unresolved problem in this patch:

// @todo this causes infinite recursion… any ideas for a work-around?
wim leers’s picture

Issue summary: View changes

Status: Needs review » Needs work

The last submitted patch, 1: header_value_size_limit-2542868-1.patch, failed testing.

wim leers’s picture

Issue summary: View changes
dawehner’s picture

+++ b/core/lib/Drupal/Core/EventSubscriber/FinishResponseSubscriber.php
index fea9762..a516607 100644
--- a/sites/default/default.services.yml

--- a/sites/default/default.services.yml
+++ b/sites/default/default.services.yml

+++ b/sites/default/default.services.yml
+++ b/sites/default/default.services.yml
@@ -1,4 +1,7 @@

@@ -1,4 +1,7 @@
 parameters:
+  # Header value size limit in bytes. -1 means infinite, no limit.
+  # @default -1
+  header_value_size_limit: -1

Can we put the default value into core.services.yml as well?

wim leers’s picture

Can we put the default value into core.services.yml as well?

Why? It doesn't need to be — Berdir and I were discussing that at the Londons sprint too: those container parameters in core.services.yml don't belong there; they belong only in services.yml (which is originally a copy of default.services.yml of course).

dawehner’s picture

+++ b/core/lib/Drupal/Core/EventSubscriber/CacheableResponseSubscriber.php
@@ -0,0 +1,131 @@
+    // When there is no limit, we never map cache tags to hashes.
+    if ($this->limit !== -1) {
+      // Also invalidate the cache tags as hashes, to automatically also work
+      // for cached responses that exceed the header size limit.
+      // @todo this causes infinite recursion… any ideas for a work-around?
+      Cache::invalidateTags(static::cacheTagsToHashes($tags));
+    }

just add a static var to detect the recursive calls?

fabianx’s picture

#6: That is a good reasoning, but the problem is the upgrade path again:

Think of it like:

variable_get('foo_variable', 'default');

And then in settings.php a:

#$settings['foo_variable'] = 'some-value';

Container parameters don't have a default, so core.services.yml _is_ our default as container parameters are overwritten by whoever comes last in $settings['container_yamls'].

If we don't have a default in core.services.yml or for a core module in module/module.services.yml then updating core versions fails as the parameter is missing.

That is why we need both IMHO.

wim leers’s picture

#7: don't we want to avoid adding state to services? But yeah, I don't have other ideas either.

paulmckibben’s picture

Re #7: I tried the static variable approach, and it seems to be working.

  /**
   * Static variable to control recursion in the invalidateTags method.
   * @var bool
   */
  private static $recursionStop = FALSE;
  /**
   * {@inheritdoc}
   */
  public function invalidateTags(array $tags) {
    // When there is no limit, we never map cache tags to hashes.
    if ($this->limit !== -1) {
      // Also invalidate the cache tags as hashes, to automatically also work
      // for cached responses that exceed the header size limit.
      if (!static::$recursionStop) {
        static::$recursionStop = TRUE;
        Cache::invalidateTags(static::cacheTagsToHashes($tags));
        static::$recursionStop = FALSE;
      }
    }
  }
sorressean’s picture

StatusFileSize
new9.15 KB
new3.38 KB

Just uploading this to see if it fails tests.

sorressean’s picture

Status: Needs work » Needs review

Go testbot go! Make me proud!

Status: Needs review » Needs work

The last submitted patch, 11: header_size_limit-2542868-11.patch, failed testing.

sorressean’s picture

StatusFileSize
new9.28 KB
new246 bytes

Here's another attempt at this.
I put the default value in core.services.yml. I'm also thinking there should be a test for this, but I'm not really sure how to make that work, so any thoughts would be great.
Thanks,

sorressean’s picture

Status: Needs work » Needs review
wim leers’s picture

but I'm not really sure how to make that work, so any thoughts would be great

Create a test route whose controller does something like this:

class Controller {
  public function cacheTagsTest($length) {
    $build = [];

    // 9 is the length of each cache tag, 1 is for the space separating two cache tags.
    $num_cache_tags = $length / (9 + 1);
    $last_cache_tag = $some_modulo_handling_here;

    for ($i = 0; $i < $num_cache_tags; $i++) {
     $build['#cache']['tags'][] = Random::name(9, TRUE)
    }

    return $build;
  }
}

That then allows you to easily generate responses with an X-Drupal-Cache-Tags header of various lengths, and in the test you can change the value of the limit to verify that it handles it correctly.

Status: Needs review » Needs work

The last submitted patch, 14: header_size_limit-2542868-14.patch, failed testing.

borisson_’s picture

Status: Needs work » Needs review
StatusFileSize
new9.28 KB

Reroll of #14 attached.

Status: Needs review » Needs work

The last submitted patch, 18: reroll.patch, failed testing.

The last submitted patch, 18: reroll.patch, failed testing.

sun’s picture

 parameters:
+  header_value_size_limit: -1

http.response.header.max_length would be a much more appropriate parameter name.

wim leers’s picture

wim leers’s picture

Status: Needs work » Closed (works as designed)

#2527126: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them was just fixed.

The solution we arrived at there is to never send the X-Drupal-Cache-Tags header except for debugging (and tests; because otherwise we would break existing tests).

Since this then exists for debugging purposes only, it is pointless to then limit the size by hashing, because that means you can no longer see the cache tags, which defeats the purpose of showing them for debugging.

Therefore marking "works as designed".

berdir’s picture

The solution we arrived at there is to never send the X-Drupal-Cache-Tags header except for debugging (and tests; because otherwise we would break existing tests).

Uhm. And actually using them for varnish etc. cache invalidation? :)

I'm not sure if we need this or not, but that argument doesn't really work ;)

moshe weitzman’s picture

@Berdir - the idea is that cache tag headers will be emitted by Cloudflare module, Varnish module, etc. Core is no longer in the business of sending production headers.

wim leers’s picture

What Moshe said. Precisely because each of those services have their own expectations regarding header name (Fastly: Surrogate-Keys, CloudFlare: Cache-Tag …) and header value format (space separated, comma-separated …).

Exactly that is mentioned in the CR too: https://www.drupal.org/node/2592471, point 2.

wim leers’s picture

In other words: even if we implement a generic setting, it'd still be up to those modules to respect it.

So… I guess we could do it, but it'd be a relatively strange API/setting/container parameter at that point: it would be mostly for things *outside* core.