Exposed in #2551419: Abstract RenderCache into a separate service that is capable of cache redirects in a non-render array-specific way so you can read the full thought process there but here's the TLDR:

Both X-Drupal-Cache-Tags and X-Drupal-Cache-Contexts are set in FinishResponseSubscriber (FRS). This is before page cache gets the response but after dynamic page cache has handled it. This is wrong for both tags and contexts.

To add insult to injury, FRS folds the cache contexts before setting them in the header, but does not similarly affect the cache tags. This is very misleading for numerous reasons:

  • Only the cache contexts gathered up until DynamicPageCacheSubscriber (DPCS) truly matter because that's where they'll affect the cache. Any contexts added later on -such as in RouteAccessResponseSubscriber (RARS)- do nothing at all because page cache doesn't leverage contexts, only tags.
  • By folding the contexts in FRS, we therefore expose incorrect data. Suppose we got to DPCS with user.permissions and not user, but RARS would add user to the response, then X-Drupal-Cache-Contexts would simply show user, even though the DPC stored the page with user.permissions
  • If we are folding contexts, it affects the tags. This is not represented in X-Drupal-Cache-Tags

The reason this currently works with our thousands of tests checking for the 2 headers is twofold:

  1. RenderCache (accidentally?) bleeds information about folded contexts/tags, which causes it to bubble up to the response. By rewriting RenderCache in the issue linked at the top, this became very obvious through many test fails.
  2. Most of our tests check for the wrong contexts because their writers seemingly had no clue how broken the headers were. We even have tests checking for user after folding user.permissions, but not the tags set by said folding.

Proposed solution

In the long run, we should move X-Drupal-Cache-Tags to PageCache, where it actually makes sense. We should also move X-Drupal-Cache-Contexts to DPCS where it makes sense for contexts to be debugged.

This will, however lead to ambiguity in X-Drupal-Cache-Tags because we might want to debug tags for DPC and PC individually. So I suggest we add a third header X-Drupal-Dynamic-Cache-Tags and rename the context one to X-Drupal-Dynamic-Cache-Contexts.

We can finally add X-Drupal-Cache-Contexts-Deprecated to PageCache and allow failing tests to check that header while we clean them up on a smaller scale in follow-ups. The end goal would be to remove said header altogether.

Comments

kristiaanvandeneynde created an issue. See original summary.

kristiaanvandeneynde’s picture

Status: Active » Needs review
StatusFileSize
new8.77 KB

Step 1: Let's introduce new headers and leave the old ones untouched.

The only major difference here is that we now affect cacheability in DPCS by folding contexts for all responses, not just the ones that came from the renderer.

After we fix failures caused by this, we can go to step 2: Move all usages of X-Drupal-Cache-Contexts to X-Drupal-Dynamic-Cache-Contexts and see how many failures we get. Then we can fix those by allowing them to use the old header and create follow-ups to evaluate why they were failing and fix them.

berdir’s picture

One thing to consider is what the purpose of those headers is, and as far as core is concerned, it is *only* for debugging/testing.

It's not about what is considered by our cache implementations but what the final page, as a whole, varies on. You could implement a page cache that does consider cache contexts, the header is not there to tell you about what the dynamic page cache included. It's X-Drupal-Cache-Contexts, not X-DPC-Contexts.

kristiaanvandeneynde’s picture

Which is why I think we need to move the headers to the actual caches. It makes no sense for FinishResponseSubscriber to set headers because said subscriber does not handle caching and doesn't know how the caches use the cacheability. If we move them to DPC and PC respectively, then we can make sure the headers actually reflect what is happening in the cache.

If you then want to replace page cache to respect contexts, you could output your own headers if you choose.

The fact that they're used for debugging only is nice, but if an upgrade of RenderCache causes the info to actually be correct and said change causes a lot of test failures because our tests relied on the wrong info, then what's the point? Tests checking for both user and user.permissions completely beats the purpose of testing as it's an unrealistic scenario to begin with.

kristiaanvandeneynde’s picture

StatusFileSize
new1.03 KB
new7.96 KB

Test will fail massively, something extra slipped into the patch -.-

Status: Needs review » Needs work

The last submitted patch, 5: drupal-3052395-5.patch, failed testing. View results

kristiaanvandeneynde’s picture

Status: Needs work » Needs review
StatusFileSize
new7.47 KB
new3.59 KB

Okay so this surfaced a few tests that fail when cache context folding does affect tags. Now, I've been giving it some though and internal cache mechanisms should never affect what you're caching, as demonstrated by these test fails. This is also reflected in the other issue's changes to RenderCache, which did alter the very thing it was caching.

Only DPC affects tags/contexts when setting the actual item in its cache. That does not mean that PC wants to work with these altered cache tags. It simply doesn't care. As proven by the need for AnonymousUserResponseSubscriber to exist. This also means the page cache debug headers should not optimize contexts as that's not what happens in its cache.

So attached is a patch that still outputs the new headers, but does not affect the response whatsoever. This should go completely green.
I'll also attach a patch that moves the page cache headers to page cache and makes sure page cache doesn't affect the response's cacheability either.

Once we have those 2 sets of headers in the right place, I can update X-Drupal-Cache-Contexts to not optimize its tokens as it provides false information to the tests/debugger. I expect tests to fail there, which we can then update.

kristiaanvandeneynde’s picture

StatusFileSize
new9.62 KB
new14.38 KB

This moves the page cache debug headers to page cache so we now have debug headers for MISS/HIT, tags and contexts in both page cache and dynamic page cache. It should go green as it still applies to all responses the same way, but I might have missed something.

This patch shows that page cache debug headers were still sent even if a request policy decided that caching should be denied:

if ($type === static::MASTER_REQUEST && $this->requestPolicy->check($request) === RequestPolicyInterface::ALLOW) {

If this line failed because of a request policy, FinishResponseSubscriber would still kick in and output the headers because FRS only cared about being on the master request and dealing with a CacheableResponseInterface respone.

I'm interested to find out how many tests would break if we only sent debug headers when the above check passed.

kristiaanvandeneynde’s picture

StatusFileSize
new5.58 KB
new16.12 KB

Now that #7 went green, let's put our money where our mouth is. Let's make sure the "should we cache this" check uses the same contexts we actually output in the header. This should be interesting.

The last submitted patch, 8: drupal-3052395-8.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 9: drupal-3052395-9.patch, failed testing. View results

wim leers’s picture

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.9 was released on November 6 and is the final full bugfix release for the Drupal 8.7.x series. Drupal 8.7.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.8.0 on December 4, 2019. (Drupal 8.8.0-beta1 is available for testing.)

Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

mstrelan’s picture

Version: 9.5.x-dev » 10.1.x-dev
Assigned: kristiaanvandeneynde » Unassigned
Issue tags: +Bug Smash Initiative, +Needs tests

Can we write a test for this? Or is this covered by existing tests?

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

moshe weitzman’s picture

Priority: Normal » Major

Boldly upping priority because these headers are a foundational part of perf improvement on all Drupal sites.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.