Problem/Motivation
As part of the system.performance
config, Drupal currently includes the cache.page.max_age
property that provides the max-age value in Cache-Control header on cacheable responses. There are many scenarios where it is desirable to have an independent max-age for 4xx responses.
Proposed resolution
In addition to invalidating the Drupal cache ( #2472281: 404/403 responses for non-existing nodes are cached in Page Cache/reverse proxy, are not invalidated when the node is created), make the emitted max-age for 4xx responses independent of the standard max-age.
This new property should still be part of the system.performance
and named cache.page.4xx_max_age
and be nullable
. The value of this new property should be set to null
initially and fallback to the standard cache.page.max_age
if not set.
Remaining tasks
Create change record. Stub: #2972025
User interface changes
No changes (per discussion in comments below).
API changes
New config to provide the max-age value in Cache-Control header on 4xx responses.
Discovered by pwolanin & klausi.
Comment | File | Size | Author |
---|---|---|---|
#135 | 404_max_age.patch | 10.58 KB | abonfatti |
#130 | interdiff_124-128.txt | 4.75 KB | SandeepSingh199 |
#128 | 2472335-128.patch | 9.22 KB | SandeepSingh199 |
#125 | 2472335-124.patch | 10.47 KB | Ratan Priya |
#120 | reroll_diff_2472335_116-120.txt | 9.33 KB | ankithashetty |
Comments
Comment #1
pwolanin CreditAttribution: pwolanin at Acquia commentedComment #2
pwolanin CreditAttribution: pwolanin at Acquia commentedComment #3
mr.baileysComment #4
Wim Leersmr.baileys is working on a different issue for now.
Comment #5
borisson_Comment #6
borisson_I added this in the FinishResponseSubscriber
Comment #8
Wim LeersLooking good!
Failed because #2368987: Move internal page caching to a module to avoid relying on config get on runtime landed in the mean time.
A bunch of nitpicks. Point 3 needs feedback from others, e.g. Fabianx. Pinged him.
s/results/responses/
Max age for 4xx responses
I personally think we don't want a UI for this. Which would mean these changes are unnecessary.
Let's see what others think.
The page cache is already enabled; this is unnecessary.
Comment #9
Fabianx CreditAttribution: Fabianx for Acquia commenteda) I don't think we want a GUI.
b) I don't see why we need this at all. After all you can just add a subscriber if you want to change the response and use a different max-age.
This needs a IS update.
Comment #10
borisson_I changed the comments wim had in #8 but left the UI in for now.
I discussed the change to fix test failures with znerol and he agreed this was a good way to do this.
Comment #11
borisson_Comment #13
Wim Leers@znerol, @pwolanin, please chime in.
Comment #14
borisson_I fixed the failures testbot failures, an unrelated test in the same testClass fails for me, but I believe this is unrelated. Let's see what testbot thinks.
Comment #17
Wim LeersPinged @znerol for a review.
Comment #18
znerol CreditAttribution: znerol commentedOpening brace should be on a separate line.
It should not be necessary to change this condition. If
isCacheControlCustomized()
acts in weird ways on different machines, then we need to fix that. Also I cannot reproduce the test failure in #10 locally. Perhaps this was just a test-bot hiccup?Comment #19
borisson_Fixed both changed suggested by znerol in #18. Don't have any test fails either locally without the changed if condition.
Comment #20
znerol CreditAttribution: znerol commentedIf we default to 0 for normal pages, we also should use 0 for error pages.
We could move the 4xx mapping one level up in the structure. After the extraction of the page cache module, the
cache
key in thesystem.performance
config has no other responsibility than governing themax_age
directive of theCache-Control
header.Comment #21
borisson_I changed the indentation of the config schema and changed all related settings in this patch. default value is now 0 as well.
I also added a new eventListener that listens to the configCrudEvent and invalidates the '4xx-response' tag. I adjusted the test to account for this as well.
Happily sent from the car between Montpellier and Belgium.
Comment #22
Wim Leers:)
Let's reduce this to a single newline.
Suggested: Invalidates the '4xx-response' cache tag if the max-age for 4xx responses was changed.
(If that fits.)
This is not true. C/p remnanet, probably.
Needs FQCN.
s/array()/[]/
Needs newline between these two.
Comment #23
borisson_Fixed coding standards as requested in #22.
Comment #24
borisson_Comment #26
znerol CreditAttribution: znerol commentedUse identity operator when comparing strings (i.e. '===').
Comment #27
borisson_I changed the code to use strict checking, i copied this from the ConfigSubscriber in language and that uses "==" as well. I also rerolled the patch to be an actual patch against HEAD, not just an interdiff.
Comment #28
znerol CreditAttribution: znerol commentedThanks, this looks good now.
Comment #29
Fabianx CreditAttribution: Fabianx for Acquia commentedMost routes are already fixed by the 4xx subscriber - if the menu router is rebuild.
I am still not sure we need it, but I am also not opposed to it. It seems like an advanced usage.
However, because the other issue has landed already:
This is not a major bug, such changing to a "normal task".
- This needs an issue summary update to show use cases, where this still breaks.
- This needs a beta evaluation.
Comment #30
catchTagging D8 upgrade path. If it's not configurable (per #9) we wouldn't need the upgrade path.
But also I have the same question as Fabianx - it's possible that something not-an-entity could invalidate a 4xx page, but then that other thing could invalidate the cache tag. By the time you're at the point where you're tweaking the value, you could just do that.
Comment #31
BerdirI think it should be configurable to actually allow to switch between setting max-age or not in the first place. We don't cache the normal max-age by default either, because doing so, especially for a long value, requires a varnish/$reverse_proxy integration module that can deal with cache tag invalidations. I think enabling that by default and enforcing it would be a bad idea. Unlike page cache (where we already cache 404's, permamently), external caches don't work out of the box.
Comment #32
borisson_Going to unassign this issue, at least for now. I'm going to let you guys decide if this is really needed or not.
Comment #33
Fabianx CreditAttribution: Fabianx commentedAssigning to catch based on the Varnish argument ...
Comment #34
catchThis is true, but it's true for the default max-age as well.
So either:
1. You have varnish + no cache tag invalidation and probably a short TTL for all responses. This works fine for some sites.
or
2. You have varnish + cache tag invalidation and a longer TTL is fine because cache tags will work.
I don't see how 403/404 responses are different/worse from 200s if they get stale - usually in terms of cache invalidation people are worried about:
1. Deleted/unpublished content still showing up.
2. Edited content not showing up.
3. New content not showing up.
And usually more or less in that order.
Comment #35
BerdirRight, yes, it's the same for the normal max-age.
So you're not really arguing against making it configurable, you're arguing against making it configurable separately from the normal page max-age?
Which would basically mean that this issue is a won't fix. That would be fine with me, especially since we now have the tools to invalidate them in a fairly reliable way. Personally, I think the ability to cache 5xx errors with a different max age would be more interesting, as those are usually temporary and can't be invalidated automatically. E.g., you could cache the response when the database is down for a short time, to avoid hundreds of requests. We currently explicitly prevent that.
Comment #37
catchComment #38
catchGeneral note that #2699613: Set a shorter TTL for 404 responses in page_cache module is changing this for page_cache module, since permament cache items in the database cache can cause disk-filling.
Still think given varnish doesn't have that problem the existing behaviour with max-age might be OK, but posting this so I don't get confused which issues is which again - already happened twice this month.
Comment #41
Wim LeersHow do we want to continue here?
Comment #42
gg4 CreditAttribution: gg4 commentedSame question. This would an incredibly useful feature to land.
Comment #43
dawehnerIs it just me, or is this something we should not expose via the UI, but just make it configurable?
Comment #44
catchAgreed with #43.
Comment #45
dawehnerThe same should be true for
'cache.page.max_age'
, right?Comment #46
borisson_#45: I don't think so, at least not for 4xx pages, the max age for those is different. If we'd want to do that for normal pages, we should do it in another issue.
I also agree with removing the UI, 2 years ago, this was one of my first core patches and I was too attached to this code to go ahead and remove it, while it was already suggested in #8 and #9.
A lot has changed in the meanwhile though, so this no longer applies.
Comment #47
borisson_Setting to needs work, because the patch no longer applies.
Comment #48
gg4 CreditAttribution: gg4 commentedAttached is a re-roll, with UI options removed. Tests might need a review because of #2699613: Set a shorter TTL for 404 responses in page_cache module, but let's give is a go.
Comment #50
gg4 CreditAttribution: gg4 commentedQuick fix of coding standards issues. Tests will still need work.
Comment #54
gg4 CreditAttribution: gg4 commentedReroll of #50.
Comment #56
dawehnerI want to ask a question whether we could do actually better for entities. In the case of entities we know its path, so what about adding the path as cache tag to page cache and then on entity save build up a path using the 'canonical' link template.
On the other hand I'm not sure whether its worth special casing entities here?
Comment #57
catchUnassigning me.
Comment #58
gg4 CreditAttribution: gg4 commentedComment #60
gg4 CreditAttribution: gg4 commentedComment #62
gg4 CreditAttribution: gg4 commentedComment #64
gg4 CreditAttribution: gg4 commentedComment #65
gg4 CreditAttribution: gg4 commented...
Comment #66
gg4 CreditAttribution: gg4 commented#65 -- patches and tests for both the UI and non-UI options for
cache.4xx.max_age
.Comment #67
borisson_Should be only one line with a max of 80 cols.
In any case, I don't think the version of the patch with the UI is needed anymore, enough people have said that it's not needed.
Comment #68
gg4 CreditAttribution: gg4 commentedEasy fix. Thanks.
Comment #69
borisson_Awesome, thanks! Even though it's been over 3 years I don't think I'm allowed to rtbc that.
Comment #70
Wim LeersThe comment is inaccurate: it applies to all 4xx responses, not just 403 and 404.
Honestly, I think this comment can simply be removed.
Nit: Suggested change:
This belongs in the
system
module, not in thepage_cache
module.Because:
system
moduleFinishResponseSubscriber
, which doesn't live in thepage_cache
module either
such as Varnish. If this subscriber only runs when Page Cache is installed,
then Varnish will continue to serve stale 4xx responses
Therefore this can be merged with
\Drupal\system\SystemConfigSubscriber
.Nit: let's make these single lines.
Why is
4xx
a key on the same level aspage
?Wouldn't
cache.page.max_age
(existing)cache.page.4xx_max_age
(new)be more appropriate names?
I don't know why this provides migrate paths. This is not something that existed in D6 or D7.
It shouldn't provide a migration path. It should provide an update path.
Because existing D8 sites still won't get this new setting.
Comment #72
gg4 CreditAttribution: gg4 commentedReroll and addressing the comments in #70. Probably could use some additional tests in the system module.
Comment #73
gg4 CreditAttribution: gg4 commentedNeeds review for the bot.
Comment #75
gg4 CreditAttribution: gg4 commentedComment #76
gg4 CreditAttribution: gg4 commentedComment #78
gg4 CreditAttribution: gg4 commentedComment #79
gg4 CreditAttribution: gg4 commentedOk. Green again. Notable changes form #68 considering #70:
cache.page.4xx_max_age
, wascache.4xx.max_age
4xx_max_age
in nownull
, was0
. Otherwise we will stop caching 4xx responses unless end users set a value for4xx_max_age
ConfigSubscriber
logic tosystem
modulecache.page.4xx_max_age
in the$expectedConfig
Comment #80
Fabianx CreditAttribution: Fabianx as a volunteer commentedChanges look very great to me. As I am not seeing anything missing, setting to RTBC.
Comment #81
alexpottThis can be $client_error_max_age - we've already got the value from config.
I think this should have something like
Before it so it is obvious we're testing that setting the value invalidates the cache.
This should also have
nullable: true
Comment #82
gg4 CreditAttribution: gg4 commentedAddresses code comments in #81.
IS update and CR still needed.
Comment #83
alexpottAs above we need to add
before this to show that the cache was HIT before this MISS.
Comment #84
gg4 CreditAttribution: gg4 commentedAdding overlooked change from #83.
Comment #85
gg4 CreditAttribution: gg4 commentedComment #86
gg4 CreditAttribution: gg4 commentedComment #87
gg4 CreditAttribution: gg4 commentedIS updated and CR drafted.
Comment #88
Fabianx CreditAttribution: Fabianx as a volunteer commentedThe change record https://www.drupal.org/node/2972025 looks good to me.
Comment #89
Fabianx CreditAttribution: Fabianx as a volunteer commentedI meant to RTBC this actually.
Comment #90
catchI still have the same reservations here as I did in #33 - I can't see what the use-case for this is. i.e. there seem to be two situations:
1. You already have max_age short because your external proxy doesn't support cache tags. If this is the case, what makes 403/404 responses special? Everything will need a short max_age.
2. If the external proxy does support cache tags, then you probably don't need to configure it separately either, because cached 403 or 404 responses will in 99.9% of cases be invalidated when content is published and similar.
If there is a #3, it would be really useful to document that both in the issue summary and the change record.
Comment #91
anavarre@catch - consider this use case:
At the reverse proxy (e.g. Varnish VCL), say you enforce 30mn caching for all 404s to prevent DDoS attacks. This means when editors check if a page exists and get a 404 because it doesn't, Varnish is going to cache the page for 30mn. If it happens that editors do want to create this page, they'll do so and will notice the page is stubbornly throwing a 404 as anon. This creates a frustrating experience and is one of the reasons why https://www.drupal.org/project/http_cache_control was created, if I understand correctly.
Whether it should be supported by default in core is a different story.
Comment #92
gg4 CreditAttribution: gg4 commentedThanks for the review, @catch.
First, Core has
$settings['cache_ttl_4xx'] = 3600;
with applies only to thepage_cache
module. This issues allows similar functionality for external proxies.Second, and a variation of your 1. in #90, you have an external proxy that doesn't support cache tags, but still want a reasonable
max_age
/ CHR. For amax_age
set to 60 min, let's say, newly published content could still 404 for an unacceptable amount of time. Being able to setcache.page.4xx_max_age
to 1 min with keepingcache.page.max_age
at a larger value still provides some baseline 4xx traffic stampede protection and quick expiration, while helping maintain a higher CHR on 200s.Comment #93
catch@anavarre I understand that use case, but it's covered by using cache tags and varnish which is already supported: https://www.drupal.org/docs/8/api/cache-api/cache-tags-varnish
@bonus
That's not why that setting exists, it's there because the default backend for the page cache is the database, and entries are stored permanently, so it was possible to DDOS a site simply by visiting 404s and fill the disk. See #2697795: [meta] Cache storage in database can fill the disk, especially via 404s.
Not caching 404s when a real page is created at the URL is handled instantaneously via cache tags - both with the database and reverse proxies that support cache tags.
So the only case would be a reverse proxy that does not allow cache tags - but again there's a limit to how long a site can have that set for non-4xx responses, you'd expect it to be pretty short and then also use the internal page cache as a backup (which is always up-to-date with the latest content, so can be cached permanently). If this is the only valid use case, I'm not sure it's worth supporting in core - would prefer to improve the documentation on the various options somewhere.
Comment #94
gg4 CreditAttribution: gg4 commented@catch, thanks for the clarification on
cache_ttl_4xx
.Comment #95
Fabianx CreditAttribution: Fabianx as a volunteer commented#90:
Given that I just implemented a backport of that for Drupal 7 for one of our enterprise clients, let me elaborate:
It is perfectly fine to have 10 min of caching for all pages, but only 20s for 403 and 404 responses.
The reason is:
- Most CDNs support a shield-like functionality, so if you don't cache 403 and 404, then all requests are send to origin.
- But you definitely don't want to cache a 404 / 403 for longer than 20s as the page could then exist and this is an error state.
e.g. the only reason you need a max-age on 404 / 403 (and not uncacheable) is the shield like functionality of CDNs / Varnish as you want micro-caching, but a user should never see an error longer than necessary.
And most Varnish configurations do not yet support cache tags out of the box and e.g. Akamai is just now updating to have some kind of Surrogate-Keys.
I hope that helps as justification.
Comment #96
effulgentsia CreditAttribution: effulgentsia at Acquia commented#95 sounds pretty reasonable. Just one question though:
Isn't 10 min of caching a 200 in a reverse proxy that doesn't support cache tags just as much of an error state as 10 min of caching a 4xx? What makes the former error more acceptable than the latter one for the client that you helped with this?
Comment #98
Fabianx CreditAttribution: Fabianx as a volunteer commented#96:
The reason why a 403 / 404 page is more of an error state is, because they are very obvious errors and also more fatal for the UX.
e.g. if a title somewhere in a sidebar is wrong, it might be not a big deal to wait for the 10 min to expire, but if for example an access check is wrong and users are getting 403 pages, then this is an error state, which should be fixed ASAP as it is very obvious to the users that the 'site is broken'. And Drupal itself also handles those pages and conditions differently (logging it, etc.).
So while I agree that technically there is no difference between an error on a normal page and an error presented by a 404/403 page, there is a large difference in user perception.
Comment #100
Fabianx CreditAttribution: Fabianx as a volunteer commentedIt passed again.
Comment #101
effulgentsia CreditAttribution: effulgentsia at Acquia commentedYep, #98 convinces me of that.
However, are there also common scenarios for wanting an independent max-age for 3xx responses and/or wanting 3xx responses to use the same max-age as 4xx responses? Just asking to make sure that
4xx_max_age
is the config variable name that we're confident we'll want to keep for a while.Comment #102
Fabianx CreditAttribution: Fabianx as a volunteer commented#101 For 30x the story is a little bit different from my experience:
Either we want to cache the 30x redirect not at all (e.g. a language redirect or based on some user session data) or we want to cache it permanently / per max-age (e.g. there is a way for it to expire) and then usually the default expiration time is fine (e.g. the same as max-age).
In all those cases the caller of "drupal_goto()" / the thrower of the RedirectResponse() is responsible for the cacheability of the actual redirect.
I have not come across the use-case in practice that we would need 3xx responses to work like 4xx and you never want to cache 5xx responses.
Therefore I would say that 4xx_max_age is a fine name and also matches what is in page_cache already as configuration variable (even if there it was a different reason).
Feels consistent.
Comment #104
larowlanWe need a post update hook here with a test, but other than that, looks good.
I'd refactor the calculation of $max_age into a protected method to avoid else, but that's a personal preference.
Comment #105
larowlanSee #2828793: Stop logging comment IP addresses by default for sample update path and test
Comment #106
mirsoft CreditAttribution: mirsoft commentedRerolled patch that is compatible with Drupal 8.6.10 (there was a duplicate `system_update_8601()` function in previous one).
Comment #108
abonfatti CreditAttribution: abonfatti commentedUpdated patch to be compatible with Drupal 8.7.7
Comment #111
godotislate CreditAttribution: godotislate commentedRe-roll of #108 against 9.1.x. Changes from #104 and #105 pending.
Comment #112
raman.b CreditAttribution: raman.b at OpenSense Labs commentedAdding post update hook and resolving failed test cases from #111
Comment #114
raman.b CreditAttribution: raman.b at OpenSense Labs commentedUnrelated test failure.
Comment #116
kiseleva.t CreditAttribution: kiseleva.t as a volunteer and at FFW commentedAdjusted patch from #112 to core 8.9.x.
Comment #118
rahulrasgon CreditAttribution: rahulrasgon at QED42 for Drupal India Association commentedRe-roll of #112 against 9.2.x.
Comment #120
ankithashettyRerolled patch in #116 and updated deprecated functions like changed
$this->assertEqual($this->drupalGetHeader(''), '')
to$this->assertSession()->responseHeaderEquals('', '')
wherever necessary, thanks!Comment #124
nod_D10 version needed
At this time we need a D10.1.x patch or MR for this issue.
Comment #125
Ratan Priya CreditAttribution: Ratan Priya at OpenSense Labs for DrupalFit commentedRerolled patch for 10.1.x.
Needs review.
Comment #127
SandeepSingh199 CreditAttribution: SandeepSingh199 as a volunteer commentedComment #128
SandeepSingh199 CreditAttribution: SandeepSingh199 as a volunteer commentedrerolled the patch for D10.1.x and fixed the issue of #125 . Please check and confirm.
Comment #129
SandeepSingh199 CreditAttribution: SandeepSingh199 as a volunteer commentedComment #130
SandeepSingh199 CreditAttribution: SandeepSingh199 as a volunteer commentedComment #131
smustgrave CreditAttribution: smustgrave at Mobomo commented1. Seems out of scope
2. Same
3. 3 more instances of spacing being added which is out of scope
and patch #129 is removing test coverage.
Comment #133
szato CreditAttribution: szato at Brainsum for The International Federation of Red Cross and Red Crescent Societies (IFRC) commentedUsing patch #124 on D 10.1.8 fixed my issue.
@smustgrave regarding extra lines in interdiff 124-128 (point 1., 2.) - an empty line was removed in 124, so these are OK in #128 I think.
Point 3. and test coverage should be fixed.
Comment #134
szato CreditAttribution: szato at Brainsum for The International Federation of Red Cross and Red Crescent Societies (IFRC) commentedI think the assignment is not relevant after more than 1 year.
Comment #135
abonfatti CreditAttribution: abonfatti commented