Problem/Motivation

If you want to return different responses for the same request, depending on values of HTTP headers, resulting response cannot be cached neither in internal nor in external response-level cache.

Examples of such requests:

- Browser language negotiation (#2430335: Browser language detection is not cache aware) should have different results depending on visitor's language preferences, sent in "Accept-Language" header.
- Site needs to display local sales office to people depending on their originating country. Many CDN providers (Cloudflare, Akamai etc.) sent this information in HTTP header.

The only existing way to return headers-dependent response is to mark it non-cacheable (use KillSwitch policy). That happens because cacheability metadata that contains "header" context (implemented in HeadersCacheContext class) is only stored in renderable arrays, but not exposed to reverse proxies (or internal page context).

Proposed resolution

Introduce new Vary page response policy, that will tell reverse proxies how to properly cache returned responses. This proposal was originally raised in #2430335: Browser language detection is not cache aware, but moved to the new issue, as it solves a bigger problem.

@catch in https://www.drupal.org/project/drupal/issues/2430335#comment-12842970 argues against usage of vary header in core itself, but with all respect, I cannot agree - Vary is the standard HTTP way of informing proxies about cache contexts of the origin server, especially when the response varies depending on one or more HTTP headers. Vary header is here exactly for this purpose, so it seems natural to use it.

Remaining tasks

- Rewrite the patch from https://www.drupal.org/project/drupal/issues/2430335#comment-12840473, leaving only the part that introduces new response policy (and it's test)
- Modify the patch in #2430335: Browser language detection is not cache aware to only contain changes needed for browser language negotiation.

User interface changes

- No UI changes

API changes

- New page_cache_vary service that triggers adding Vary header

Release notes snippet

Issue priority

Major since it unblocks major bug #2430335: Browser language detection is not cache aware

Issue fork drupal-3023104

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

valthebald created an issue. See original summary.

valthebald’s picture

Issue summary: View changes
Status: Active » Needs review
StatusFileSize
new6.68 KB
new9.51 KB

The last submitted patch, 2: 3023104-testonly.patch, failed testing. View results

znerol’s picture

Regrettably this approach clearly violates the ResponsePolicyInterface contract. While not stated explicitly in the documentation, the check() method is certainly not expected to actually change the response.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

vidorado’s picture

Rerolled patch for Drupal core 8.7.6

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

fago’s picture

> Site needs to display local sales office to people depending on their originating country. Many CDN providers (Cloudflare, Akamai etc.) sent this information in HTTP header.

Afaik this was the original path during Drupal 8 development, but it was replaced by the current approach of using URLS and the support fro the `_format` query parameters. This was done since there is only bad or hardly proper support for the Vary http headers across proxies and CDNs, so using the query parameters makes the url is different and cachable.

I guess that decision could be re-evaluated now, since a couple of years passed, but that's the status quo.

heddn’s picture

Status: Needs review » Needs work
+++ b/core/modules/page_cache/src/StackMiddleware/PageCache.php
--- /dev/null
+++ b/core/modules/page_cache/src/StackMiddleware/PageCache.php.rej

I think this file made it into the latest patch on accident.

ravi.shankar’s picture

Status: Needs work » Needs review
StatusFileSize
new8.34 KB

Rerolled patch #7 on Drupal 9.1.x and addressed comment #10.

Status: Needs review » Needs work

The last submitted patch, 11: 3023104-11.patch, failed testing. View results

heddn’s picture

Issue tags: +Novice

Tagging novice for the minor test fixes for the failures in https://www.drupal.org/pift-ci-job/1850261.

heddn’s picture

A couple of questions, not necessarily novice their answers.

  1. +++ b/core/lib/Drupal/Core/PageCache/ResponsePolicy/Vary.php
    @@ -0,0 +1,50 @@
    +  public function add($header) {
    +    if (!in_array($header, $this->vary)) {
    +      $this->vary[] = $header;
    ...
    +      $response->setVary($this->vary);
    

    setVary has a $replace flag available to it. Should we expose that on the interface for add? Which leads to the next question, do we need to have an interface for this Vary class, or is that not necessary?

  2. +++ b/core/modules/page_cache/src/StackMiddleware/PageCache.php
    @@ -334,6 +348,14 @@ protected function get(Request $request, $allow_invalid = FALSE) {
    +    $vary = $response->getVary();
    +    if ($vary) {
    

    There is a hasVary method on the response. We should use it.

Kumar Kundan’s picture

StatusFileSize
new8.43 KB
new1.26 KB

Test fixes of #11.

heddn’s picture

Status: Needs work » Needs review
Issue tags: -Novice

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

tanubansal’s picture

Tested #15 on 9.1, 'Vary' page cache response has been added

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

gung wang’s picture

Drupal core 9.2.3 and Composer 2.1.5.

I run "composer update" after add the patch into composer.json, but I got an error:

Could not apply patch! Skipping. The error was: Cannot apply patch https://www.drupal.org/files/issues/2020-10-16/3023104-15_0.patch                                                                                                                     
  [Exception]                                                                                                                               
  Cannot apply patch Introduce Vary page cache response policy for D9 (https://www.drupal.org/files/issues/2020-10-16/3023104-15_0.patch)!  

In my composer.json file

"drupal/core": {
    ... ...,
   "Introduce Vary page cache response policy for D9": "https://www.drupal.org/files/issues/2020-10-16/3023104-15_0.patch"
}

Thank you very much!

ranjith_kumar_k_u’s picture

StatusFileSize
new8.5 KB

Re-rolled #15 for 9.3

gauravvvv’s picture

StatusFileSize
new8.04 KB
new500 bytes

Re-rolled patch #21, Fixed custom command failed. Added interdiff for same.

Status: Needs review » Needs work

The last submitted patch, 22: 3023104-22.patch, failed testing. View results

vsujeetkumar’s picture

Status: Needs work » Needs review
StatusFileSize
new8.06 KB
new979 bytes

Fixed the fail test, Please have a look.

longwave’s picture

Status: Needs review » Needs work

Questions/comments in #14 still not addressed.

znerol’s picture

As the author of the

ResponsePolicyInterface

, I'd like to point out that #4 is not addressed neither. This approach is invalid, please try to find a different extension point to solve the issue.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

nevergone’s picture

ravi.shankar’s picture

StatusFileSize
new8.05 KB
new871 bytes

Addressed point #14.2 of comment #14, keeping the status needs work for comment #14.1.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

nitesh624’s picture

Any update on this issue?
Will this be included in core?

bbrala’s picture

#2430335: Browser language detection is not cache aware has been updated to use a responsesubscriber that allows setting Vary headers. Wonder if those changes also need partial split into this issue?

dpagini’s picture

From #4...

Regrettably this approach clearly violates the ResponsePolicyInterface contract. While not stated explicitly in the documentation, the check() method is certainly not expected to actually change the response.

I don't think this is resolved with the current approaches still, as suggested in #26.
I have solved this in my own project with an event subscriber for KernelEvents::RESPONSE. I think that would be a relatively simple change to make here. Would that address the concerns of #4?

dpagini’s picture

Sorry @bbrala - this is nearly the same thing you are suggesting as well... but I do think that issue #2430335 would do most of what this issue is trying to do from what I can see.

elendev’s picture

As commented in https://www.drupal.org/project/drupal/issues/2972483, I've developed the module page_cache_vary until the issue is fixed in the core page_cache module.

It can retrieve cache vary headers by caching them per URL, so that the cost of computing the vary headers is as minimal as possible.

If this solution is good enough, I can try to integrate it directly in Drupal, let me know what you think.

bbrala’s picture

Talked with catch avout a possible angle of attack which would make this viable (possobel) from a variation and perdormance perspective. Will post that sometime soon.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.