Optimize render caching

To be able to make a proper decision I think we need proper profiling for a real world scenario. /me is starring at berdir.

Just looked at a magazine site we deployed a few days ago:

* render bin has 4 x the size of the dynamic page cache bin, and has > 90% hit rate
* dynamic page cache bin stays around 65% hit rate.
* qps rate on render bin is about 30 x qps rate on dynamic page cache
* EDIT: after 10 days since last flush, the render cache holds 500k items, while the DPC bin has 55k only

Regrettably, I have no idea of the proportion of queries to the render bin being for the page cache vs the rest of the render cache: memcache doesn't allow dump of stats beyond 1 MB per slab, AFAICS. Any suggestion to split the internal page cache from the rest ? A quick scan of the slabs suggests most of the entries are either page cache items or entity view items.

I'd think we want to keep render caching of menus, footers and other site-level items since they could potentially be built millions of times even with dynamic_page_cache, but cached per-role/language.

Entity views are much less of an issue with dynamic_page_cache though, especially the full view mode since that's almost always displayed-per-page.

There are some which are much tricker - individual comment renders on this issue will have a much higher cache hit-rate than the dynamic_page_cache, even if they're only rendered in one context - i.e. if this was 8.x each comment would have been rendered once (excluding edits) but the page will have been rendered every time a new comment is posted, which has been more than one per hour. Similar with 'teaser' type view modes in listings and similar.

Lots of trade-offs.

Yet another option, disable render caching when the database cache is used? Then it can still be on for memcache.

Interesting proposal! Maybe we disable it and then add back a few targetted use cases when it makes sense.

Added issue about splitting the page cache at #2889603: Split the internal page cache from the rest of the render cache.

Didn't see this issue yet, just staring at me unfortunately doesn't make we aware of something, especially not if the staring happens virtually :)

Moving the page cache away seems like a good idea.

Not sure yet what we should keep render cached and what not, but as discussed with Wim already a while ago, there are definitely parts that aren't worth caching. local actions/local tasks/breadcrumb blocks are a typical example that result a huge amount of variations on most sites but most likely aren't worth caching as they over time end up being by url/route and therefore the same as dynamic page cache.

The thing is that we don't really have a mechanism for that at the moment. max-age 0 is not what you want because that will mark it as uncacheable and will be placeholdered. We want something like CACHE_NOT_WORTH_IT. Similar to the never-finished issue of a min-age, that could be something that doesn't bubble up but just applies to its own render element/level. That would basically be the same as removing the cache keys, but the problem with that is that if someone removes the cache keys and then e.g. a hook_entity_build_defaults_alter() adds a key, it gets weird.

I think I would prefer to introduce such a mechanism and disable render caching where we know it's not worth it. There are many examples where it is definitely still very useful, the example from catch with a long list of comments, my standard example is always a news site frontpage (https://md-systems.github.io/drupal-8-caching/#/invalidation, the slides below that).

Another thing is that we could extend the render cache with something similar as auto_placeholder_conditions, but for things that we would not write to cache. Then we could add url/route cache contexts there, so all kinds of blocks/things that vary by route/url would no longer be cached separately. For contexts that bubble up, we would still end up doing a cache get but then we'd simply not save.

We had some major issues with cache_render on multiple projects. Drupal 8 in default configuration generates just way too many entries in cache_render table and most of it isn't used at all since the invalidation rules will invalidate multiple cache_render entries at once. So anything that is url specific is a big problem for sites with mainly anonymous traffic.

We were left with adding lots of storage to our database and frequently scanning the usage of the cache_render table - downloading 20-40GBs of cache_render just to scan it on a local machine. The biggest issue there is that we need the cache_render bin since it holds the page cache, hopefully that changes in the future, once that's done moving cache_render onto memcached becomes an option. A nice side effect on this is due to the reduced IO (saving those caches costs!) the overall page load time for the first hit becomes much better as well.

Anyways going forward we killed most of the redundant cache entries from our cache_render bin by using a module called cache_split https://github.com/derhasi/cache_split/ this adds overhead to the general cache gets and sets though - and with more rules to be inserted becomes slower. After cleaning up / identifying the parts worth keeping the render cache is actual usable for us - since there are some elements that are used across multiple pages. That cache_split module suggests using null cache backend - which I disagree with, memorybackend is a better alternative.

Relating this one #2877045: Add an option to pass through caching for instances implementing CacheableDependencyInterface which seems to have a need for @Berdir's suggested CACHE_NOT_WORTH_IT in #11. Defining very granular elements which are fine to be cached inside bigger parent elements but not being cached as themselves would be helpful. Thinking of something like Cache::PARENT.

Imo the render cache is still an important and helpful component of Drupal core, but there needs to be an easy way for being able to reduce cache granularity of the render cache - either by ['#cache'] definitions and/or by configuration.

(Probably offtopic, sorry) Has maybe someone already tried using the filesystem as backend for the render cache (not using cache.backend.php of course)? In theory, this might be an option for sites running on providers with small databases but enough file storage.

@mxh This might work for smaller systems, but once the site needs to use multiple web-servers, the file-level cache can not work consistently.

Few random thoughts.

I wonder if CacheBackendInterface needs an isLRU() method on it, or maybe getStrategy() with some constants that we define. Then core could decide whether or not to cache something depending on LRU/strategy. Perhaps render caching only make sense on LRU or something that sanely prevents runaway growth or braindead strategies like the one APC uses.

I also wonder if instead of CACHE_NOT_WORTH_IT, we have should have a #importance key, that maybe goes from 0-10, and defaults to 5. The importance factor could then be used somehow. Maybe exposed as config 'dont_cache_below_this_importance', which also defaults to 5. And then #importance could also be used for purging. Also allow it to bubble up, so the parent #importance is the max of its children's #importance.

Similarly, we could do something with the size of what is being cached.

A cache context to actually prevent caching might be an option too, but not quite sure how that would work.

What I mean is something else, see #2232375-54: Make language switcher block cacheable.

I tried to come up with a proof of concept but the problem is that the auto-placeholder conditions are used for two things.. for actually deciding when to use auto-placeholder *and* to disable dynamic-page-cache completely on those.

What I'd like is a (new) setting that triggers auto-placeholdering but wouldn't disable dynamic_page_cache and then also prevents to actually write cache entries (maybe that should also be done for normal auto-placeholdered entries?). Then we would enable that for example for url.path and url.query_args.. so for example a block that depends on that would be auto-placeholdered, no longer cached *but* we don't want to disable dynamic_page_cache just because something else, for example the main content block depends on url.query_args.

Thoughts? One option would be to add a new setting just for the logic in \Drupal\dynamic_page_cache\EventSubscriber\DynamicPageCacheSubscriber::shouldCacheResponse() but not sure if that would be considered a BC break if someone customized the other setting?

I'm sure this isn't perfect but it does seem to go in the direction I want when testing with #2232375: Make language switcher block cacheable.

Basic D8 installation with 2 languages and the language switcher block in sidebar.

On /user/ID
dynamic page cache: miss then hit
dynamic page cache has no url.query_args cache context (only url_query_args:_wrapper_format)
language block is auto-placeholdered and *not* cached

On /frontpage
dynamic page cache: miss then hit
dynamic page cache has url.query_args cache context because of the frontpage view.
language block is auto-placeholdered and *not* cached

If others think that this change would make sense then I'll open a separate issue. Also expecting test fails because I assume there are explicit tests that placeholders are render cached. Also some of my changes are likely not correct.

@wim-leers

That's now solved in #2526150: Database cache bins allow unlimited growth: cache DB tables of gigabytes!.

not solved- just hidden. It fixed the symptom not the cause. There's still IO involved without getting a real benefit from it- finding those bottlenecks is just harder now.

@Wim Leers I've seen performance increases of more than 30% due to avoiding of cache writes for unused caches. Since CPUs do not use a MySQL-Database as a Backend it probably doesn't matter that much over there ;)

btw. the performance impact of such a change with a memcached backend for instance is much lower - nature of things.

This one is something related.

Created #2935804: Add cache context exclusion list to RenderCache::set()

I'd swear we have a pre-existing issue for the whole "not worth caching" thing that is being discussed here and that is sort of taking over this issue. But having scanned #2429287: [meta] Finalize the cache contexts API & DX/usage, enable a leap forward in performance, I can't find anything :(

Is it #2241377: [meta] Profile/rationalise cache tags you had in the back of your head?

That is why I wanted an alter hook on block plugins [nutch nutch - not giving up ...], so they can have a say that they don't want to be cached (e.g. they can remove the #cache keys before being rendered), however contrary to what I might have said before:

- "no-cache" would not bubble, but it is a one-time instance, but it is hard to understand. Maybe drupal-do-not-cache-this is a better tag name.

- Generally the cache hierarchy is needed to avoid re-doing work again and again. If we don't use cache_block, but cache_render that is the same mistake as cache_page being in cache_render, the same for entities cache_render_node, etc. might be better. => e.g. lets have more granularirty. Smaller tables are better.

And that that the approach of cache hierarchy works we can see in e.g. the umami theme, where dynamic_page_cache is UNCACHEABLE for many pages.

Also list-invalidation is a big problem for DPC.

Much of the concerns above are related to MySQL as backend, but larger sites should not be using MySQL as cache backend anyway. Those could indeed use local filesystem OR SQLite3, which is blazingly fast even with pretty HUGE databases (yes it is possible just to put cache backends on SQLite3 backend).

- No idea how the MySQL issue was solved, but I found that we implemented cache tags wrongly (in an inefficient way). By using timestamps (as the checksum in the schema) we can with just one query cleanup 90% of old data in MySQL (e.g. all the expired items). That maps much more to last-modified and is an industry best-practice in HTTP anyway.

e.g. the big problem is that we never cleanup, not that cache items do not expire. We rather expire too much right now, e.g. for listings.

And while this is a generic brainstorming issue:

- I think there should be an drupal-auto-placeholder cache tag so that you don't need to specify #create_placeholder, but can also do it easily from lower levels, e.g. a block as it can't access it's final #cache properties directly ;).

- I think there should be more lazy builders used in general, but potentially need to be able to blacklist that I don't want this e.g. big_pipe'd. (new proposal: ['#cache']['blacklist_placeholder_strategies'] = ['big_pipe'];)

- We still need to implement the CachedPlaceholderStrategy (Did I create an issue for that? I can't find it right now ...)

- We can cache some things for sites where granularity in the server cache is not an option, in localStorage in the browser instead

Can we have new statistics from cache_render now that page cache was moved? cc @fgm

By using timestamps (as the checksum in the schema) we can with just one query cleanup 90% of old data in MySQL (e.g. all the expired items).

This procedure during garbage collection currently kills our sites with huge page cache tables because of locking. Instead we switched to Gzip compression, which has shown that it is very efficient. Take a look at Andre's module Compressed Cache.

This procedure during garbage collection currently kills our sites with huge page cache tables because of locking. Instead we switched to Gzip compression, which has shown that it is very efficient. Take a look at Andre's module Compressed Cache.

I have questions to that:

- Which garbage collection? Are you using temporary (e.g. max-age != PERMANENT items)? This is more like a ban-crawler and would not affect that many items at once.

- What kinda locking is happening?

- Why are you not utilizing memcache / redis for such large installations?

Compression can now be useful, because D8 is - thanks to PHP7 no longer CPU bound, but again a little bit DB-bound, but usually the difference is only for pretty large items feel-able (e.g. 10 MB data structures) in a per-item comparison.

- Which garbage collection? Are you using temporary (e.g. max-age != PERMANENT items)? This is more like a ban-crawler and would not affect that many items at once.
- What kinda locking is happening?

Drupal\Core\Cache\DatabaseBackend::garbageCollection() performs a delete statement regards expired items. It doesn't matter whether the items are expired or not. That statement is expensive and locks rows which others cannot access at this time. This is problematic especially when the table has a ton of rows. We then get a lot of lock timeout exceptions when this statement is being run, e.g.

Uncaught PHP Exception Drupal\Core\Database\DatabaseExceptionWrapper: "SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction: INSERT INTO {cache_*}

- Why are you not utilizing memcache / redis for such large installations?

We currently have to deal with a limited set of capabilities. It's also a matter of economic considerations when looking at the projects and decisions what solutions to use. Technically - of course - this makes sense at all when using Drupal 8 at these sizes, but not always economically.
Actually we currently wouldn't use a small Memcache for Page Cache at all, since we'd want as many existing Page Cache items as possible, not being thrown out via LRU because of other busy cache bins like render or data caches.

Compression can now be useful, because D8 is - thanks to PHP7 no longer CPU bound, but again a little bit DB-bound, but usually the difference is only for pretty large items feel-able (e.g. 10 MB data structures) in a per-item comparison.

With compression enabled we now have doubled our storage capabilities for render, data, page and entity cache, without sacrificing noticable performance (have a look at some results here).

#36 Thanks for the elaboration:

So you are saying the:

DELETE FROM cache_X WHERE expired <> -1 AND expired < %d

is giving you trouble even if you don't have any max-age -1 items?

Can you try to just add a SELECT in front of it to see if any rows would be affected?

The other thing that I read in a very old thread is to LIMIT the amount of data removed per item.

e.g.

do GC in the following way:

WHILE (SELECT ) {
  DELETE ORDER By expire LIMIT $limit;
}

which is recommended anyway and a major bug IMHO, because with an unbounded query it needs to do a full index scan, before returning control, while with LIMIT other processes can make progress in between the deletes.

(A limit has replication problems though and we need to order on expire in that case, too: http://mysql.rjweb.org/doc.php/deletebig)

Can you change that and report back if your deadlocks go away? (even with just one SELECT check at the beginning)

Another thing is that caches should not be inserted while being in a transaction (that is the cache consistent issue), I believe if we changed that to a simplified version of cache_consistent, we could win a lot (need to post on other issue):

While in a transaction:

- cache_get: Allow - if cache_prefix or cache tag had not been invalidated in this request [use simplified memcache key prefix algo, we could also be more strict and do invalidation by bin]
(else we run 100% cache-less, which is not really what we want even though most things are static caches anyway).

- cache_set: Disallow, treat like a record_cache_clear_all($cid), record
- cache_clear_all: Disallow, treat like a record_cache_clear_all($bin, $prefix)
- tag invalidation: Disallow, treat like record_cache_tag_invalidation

After the transaction is committed => commit all the cache clears, which is simple as just need to execute it.

That will solve the cache inconsistency for memcache / redis AND also reduce lock wait timeout exceeded problems within transactions as there will be no writes to the cache bins during the transaction.

Thanks very much @Fabianx for sharing your thoughts and suggestions about this topic, appreciate this a lot.

Can you change that and report back if your deadlocks go away?

Unfortunately this is not that easy to elaborate. We'll try to find some time for testing this (in a simple way).

So you are saying the:

DELETE FROM cache_X WHERE expired <> -1 AND expired < %d
is giving you trouble even if you don't have any max-age -1 items?

Yes, this is the affected statement, produced by DatabaseBackend::garbageCollection. As from our observations noted in #36 though, it seems that it doesn't matter what the expires value is. What does seem to matter is the huge amount of existing records in the table. I'm not an expert regards the inner guts of MySQL, but it seems that the created index on the expires column can't save us from deadlocks when big tables are in the game.

[...] because with an unbounded query it needs to do a full index scan, before returning control [...]

Seems to conform with our deadlock observations noted above.

Another thing is that caches should not be inserted while being in a transaction (that is the cache consistent issue), I believe if we changed that to a simplified version of cache_consistent, we could win a lot (need to post on other issue)

Agree. Could you post a link to that issue here if you have one, please?
Regards that lock timeout problems I recently thought about an approach based on a Drupal state variable:

1. Set the state variable "cache_rebuild" to true
2. do the cache rebuild stuff including garbage collection
-> while "cache_rebuild" is set to true, Drupal would let any attempt of cache writes run into a Null backend
3. When cache rebuild process is complete, state variable "cache_rebuild" is set to false so that Drupal "allows" cache writes again

#37

DELETE FROM cache_X WHERE expired <> -1 AND expired < %d

Gives us troubles because it needs to read the whole table ignoring any indexes:

EXPLAIN DELETE FROM cache_page WHERE expire <> -1 AND expire < UNIX_TIMESTAMP()

Result.

Using where - no sign of using index, leading to a full table scan. having a couple of gb worth of data this locked the table for good. btw. this happens also for

EXPLAIN DELETE FROM cache_page WHERE cid = 'someurl:html'
also states Using where. So I am not sure if that information is trustworthy at all.

re select statement:
EXPLAIN SELECT cid FROM cache_page WHERE expire <> -1 AND expire < UNIX_TIMESTAMP()

the select itself does not cause any problems.

this happens on a system with
innodb_version: 5.6.32-78.1

re:

WHILE (SELECT ) {
  DELETE ORDER By expire LIMIT $limit;
}

this should / will reduce the time taken for the locking of that table - given a small limit such as 100-1000. Following up on http://mysql.rjweb.org/doc.php/deletebig adding a sleep 1 in the loop might be a good idea.

#39 We are getting off-topic here, but at least for unlimited storage (e.g. not using the row limit for the DB table) this clearly shows that the temporary items make trouble - as we don't have many in core anymore anyway.

- Probably might be good to move those to a cache_page_temporary table?

On the other hand a secure and fast and scalable deletion method is needed, because in the future we should add some kind of GC for cache tagged data.

One potential - albeit crazy solution - would be to use three tables:

- cache_page and cache_page_archive

Every X hours / days, cache_page_old is REMOVED, cache_page_archive is renamed to cache_page_old, cache_page is renamed to cache_page_archive

- An item not found in cache_page is tried to be found in cache_page_archive and if it is found and valid, then it is re-written to cache_page (chained refresh hit).

- While there is more effort for that, we only keep active items in the cache, tables never get fragmented

----

Another idea for GC would be to: Just set the:

- checksum => -1 (invalid)
- data => NULL

to save space in the table, but not run into the deletion problem. Eventually few rows of invalid items should be cleaned up however, but then that could be ordered by cid, checksum.

I'll ask our database specialist today about potential solutions for the whole deletion thingy ...

#40 Reducing the occupied space of the table itself is a big performance gain (which is reflected in the garbage collection as well). See https://www.drupal.org/project/drupal/issues/1281408 and https://www.drupal.org/project/compressed_cache

I spoke with @nnewton about it and while we are not yet sure about the performance characteristics of different DELETE methods (and chunking still might make sense), we strongly advise to set the transaction isolation level to 'READ-COMMITTED', because with 'REPEATABLE-READ' you have lots of gap locking problems, which would explain your deadlocks.

This may be off topic here but we are still seeing issues with very large database cache even running 8.5.3.

We were told my our hosting team that our database cache's were getting huge. We had just done some backups and part of that process runs a cache clear so the database cache was cleared. Within 24 hours we noticed a 500MB size database cache! I checked Google Analytics to see if there was some traffic spike but there were only 600 users and a couple thousand page views - small volume traffic. I will also say that we are running varnish.

So we made changes to the settings.php file as was recommended in several articles we read even though the articles mentioned that this issue had been fixed in 8.4 We tried this anyways. These are a few of the changes we made as an experiment

$settings['database_cache_max_rows']['bins']['page'] = 500;
$settings['database_cache_max_rows']['bins']['dynamic_page_cache'] = 500;
$settings['database_cache_max_rows']['bins']['render'] = 1000;
$settings['database_cache_max_rows']['bins']['config'] = 500;

We cleared the cache on the server and saw the database cache clear out - it wasn't at 0 but it was very low. After implementing this change in the settings.php file, after just 20 minutes of normal site usage we saw the following results:

+--------------------------+--------------+------------+
| Table | Num. of Rows | Size in MB |
+--------------------------+--------------+------------+
| cache_render | 7110 | 56.16 |
| cache_menu | 692 | 20.80 |
| cache_data | 2314 | 14.11 |
| cache_entity | 1192 | 11.69 |
| cache_dynamic_page_cache | 125 | 11.67 |
| cache_page | 67 | 11.36 |
| cache_default | 212 | 3.55 |
| cache_config | 1656 | 2.77 |
| cache_discovery | 127 | 2.55 |
| cache_container | 2 | 1.53 |
+--------------------------+--------------+------------+

Notice that cache_render is at 7110 rows when the settings.php file says to limit it to 1000 rows.

Does anyone see a syntax problem in the settings.php file or is there some other files in the drupal install that might be overriding this change that we can check?

We have been trying to resolve this for three days now with no effect.

The settings are just about purging existing cached data. They do not and do not intend to prevent data from being stored in the first place. So yes, it will grew above that and will be deleted again afterwards.

It doesn't make sense to set values *that* low IMHO, then you could just as well disable dynamic page cache or set the cache backend to null. It also doesn't make sense to limit config. That won't grow above those values, it's just how many configs you have + translations and so on. But it's very little data.

You will need to investigate which parts of your site result in that many variations for the render cache, e.g. by looking at the first N characters of the cache_render table, look for the mots frequents one and then how many variations they have.

Agree with @Berdir here, you should investigate what's causing the large cache rows in the first place. I discovered recently that with search_api_tag caching on our search api views, each cached entry was around 5MB and was completely redundant as the data varied on search keys, which were also represented in the URL so the whole page cache varied on the same thing. Not all caching is good caching :)

Updating the title and adding the needs issue summary tag.

This is not about removing render caching, that's definitely still useful IMHO, it's about optimizing it, with issues like #2935804: Add cache context exclusion list to RenderCache::set(), so we for example have a way to not cache certain elements anymore in regular render cache.

Small update:

Now that we have VariationCache and made sure that nothing in core still (ab)uses the render cache to store things other than render arrays, it could be easier to remove the render cache. Having said that, I still see value in keeping it. The comments also seem to be in favor of keeping it for some parts, so perhaps we could update the IS to strike out the part that suggests outright removing the render cache?

Having said that, I think it's hard to determine beforehand what we may or may not want to exclude. For instance the IS suggests disabling it for entities and blocks but keeping it for views. But what if I have a block in my footer that is really expensive to calculate and only ever varies by what day it is, I'd rather that block be render cached and every page request from thereon out gets the cached copy than have to recalculate said block on every page request for it to then be cached by DPC.

I really like @Berdir's suggestion in #11 but I'd turn that upside down. What if we do not cache anything in render cache unless we specify that it's expensive to calculate. This is a lot safer when you consider outside alters: If I flag something as CACHE_NOT_WORTH_IT and then someone alters my render array and adds in something expensive, we have a problem. But if my render array is by default considered as not worth caching and then someone alters in something expensive, adding RENDER_CACHE_REQUIRED, then that would make more sense.

Figuring out what we want cacheable or not by default is still possible that way, we'd only need to add RENDER_CACHE_REQUIRED to the things we know are often expensive.

#59 sounds promising, also seems like it would fit quite well with lazy builders/autoplaceholdering where you have to explicitly define something in a way it can be placeholdered.