Exposed in #2551419: Abstract RenderCache into a separate service that is capable of cache redirects in a non-render array-specific way so you can read the full thought process there but here's the TLDR:
Both X-Drupal-Cache-Tags and X-Drupal-Cache-Contexts are set in FinishResponseSubscriber (FRS). This is before page cache gets the response but after dynamic page cache has handled it. This is wrong for both tags and contexts.
To add insult to injury, FRS folds the cache contexts before setting them in the header, but does not similarly affect the cache tags. This is very misleading for numerous reasons:
- Only the cache contexts gathered up until DynamicPageCacheSubscriber (DPCS) truly matter because that's where they'll affect the cache. Any contexts added later on -such as in RouteAccessResponseSubscriber (RARS)- do nothing at all because page cache doesn't leverage contexts, only tags.
- By folding the contexts in FRS, we therefore expose incorrect data. Suppose we got to DPCS with
user.permissionsand notuser, but RARS would adduserto the response, thenX-Drupal-Cache-Contextswould simply showuser, even though the DPC stored the page withuser.permissions - If we are folding contexts, it affects the tags. This is not represented in
X-Drupal-Cache-Tags
The reason this currently works with our thousands of tests checking for the 2 headers is twofold:
- RenderCache (accidentally?) bleeds information about folded contexts/tags, which causes it to bubble up to the response. By rewriting RenderCache in the issue linked at the top, this became very obvious through many test fails.
- Most of our tests check for the wrong contexts because their writers seemingly had no clue how broken the headers were. We even have tests checking for
userafter foldinguser.permissions, but not the tags set by said folding.
Proposed solution
In the long run, we should move X-Drupal-Cache-Tags to PageCache, where it actually makes sense. We should also move X-Drupal-Cache-Contexts to DPCS where it makes sense for contexts to be debugged.
This will, however lead to ambiguity in X-Drupal-Cache-Tags because we might want to debug tags for DPC and PC individually. So I suggest we add a third header X-Drupal-Dynamic-Cache-Tags and rename the context one to X-Drupal-Dynamic-Cache-Contexts.
We can finally add X-Drupal-Cache-Contexts-Deprecated to PageCache and allow failing tests to check that header while we clean them up on a smaller scale in follow-ups. The end goal would be to remove said header altogether.
| Comment | File | Size | Author |
|---|---|---|---|
| #9 | drupal-3052395-9.patch | 16.12 KB | kristiaanvandeneynde |
| #9 | interdiff-8-9.txt | 5.58 KB | kristiaanvandeneynde |
Comments
Comment #2
kristiaanvandeneyndeStep 1: Let's introduce new headers and leave the old ones untouched.
The only major difference here is that we now affect cacheability in DPCS by folding contexts for all responses, not just the ones that came from the renderer.
After we fix failures caused by this, we can go to step 2: Move all usages of
X-Drupal-Cache-ContextstoX-Drupal-Dynamic-Cache-Contextsand see how many failures we get. Then we can fix those by allowing them to use the old header and create follow-ups to evaluate why they were failing and fix them.Comment #3
berdirOne thing to consider is what the purpose of those headers is, and as far as core is concerned, it is *only* for debugging/testing.
It's not about what is considered by our cache implementations but what the final page, as a whole, varies on. You could implement a page cache that does consider cache contexts, the header is not there to tell you about what the dynamic page cache included. It's X-Drupal-Cache-Contexts, not X-DPC-Contexts.
Comment #4
kristiaanvandeneyndeWhich is why I think we need to move the headers to the actual caches. It makes no sense for FinishResponseSubscriber to set headers because said subscriber does not handle caching and doesn't know how the caches use the cacheability. If we move them to DPC and PC respectively, then we can make sure the headers actually reflect what is happening in the cache.
If you then want to replace page cache to respect contexts, you could output your own headers if you choose.
The fact that they're used for debugging only is nice, but if an upgrade of RenderCache causes the info to actually be correct and said change causes a lot of test failures because our tests relied on the wrong info, then what's the point? Tests checking for both
useranduser.permissionscompletely beats the purpose of testing as it's an unrealistic scenario to begin with.Comment #5
kristiaanvandeneyndeTest will fail massively, something extra slipped into the patch -.-
Comment #7
kristiaanvandeneyndeOkay so this surfaced a few tests that fail when cache context folding does affect tags. Now, I've been giving it some though and internal cache mechanisms should never affect what you're caching, as demonstrated by these test fails. This is also reflected in the other issue's changes to RenderCache, which did alter the very thing it was caching.
Only DPC affects tags/contexts when setting the actual item in its cache. That does not mean that PC wants to work with these altered cache tags. It simply doesn't care. As proven by the need for AnonymousUserResponseSubscriber to exist. This also means the page cache debug headers should not optimize contexts as that's not what happens in its cache.
So attached is a patch that still outputs the new headers, but does not affect the response whatsoever. This should go completely green.
I'll also attach a patch that moves the page cache headers to page cache and makes sure page cache doesn't affect the response's cacheability either.
Once we have those 2 sets of headers in the right place, I can update X-Drupal-Cache-Contexts to not optimize its tokens as it provides false information to the tests/debugger. I expect tests to fail there, which we can then update.
Comment #8
kristiaanvandeneyndeThis moves the page cache debug headers to page cache so we now have debug headers for MISS/HIT, tags and contexts in both page cache and dynamic page cache. It should go green as it still applies to all responses the same way, but I might have missed something.
This patch shows that page cache debug headers were still sent even if a request policy decided that caching should be denied:
If this line failed because of a request policy, FinishResponseSubscriber would still kick in and output the headers because FRS only cared about being on the master request and dealing with a CacheableResponseInterface respone.
I'm interested to find out how many tests would break if we only sent debug headers when the above check passed.
Comment #9
kristiaanvandeneyndeNow that #7 went green, let's put our money where our mouth is. Let's make sure the "should we cache this" check uses the same contexts we actually output in the header. This should be interesting.
Comment #12
wim leersI raised the same concerns as @Berdir over at #2551419-136: Abstract RenderCache into a separate service that is capable of cache redirects in a non-render array-specific way, with some more detail.
Comment #19
mstrelan commentedCan we write a test for this? Or is this covered by existing tests?
Comment #21
moshe weitzman commentedBoldly upping priority because these headers are a foundational part of perf improvement on all Drupal sites.