Findings from profiling a full cache clear on a relatively large site

Problem/Motivation

I did profile a cache clear (through the UI) with blackfire to see how and where we spend time on a cold cache on a larger site, this is only the data of the "Clear caches" POST request, the second half of the cost of building the first page afterwards it failed to process. Not sharing the full run for now as there's potentially internal stuff in there

This is still on 11.3, some findings may no longer apply. Using redis 2.x. Numbers taken on a a minimal dev environment on upsun, not production resources. It's not really about total numbers, just seeing what's going on.

Overall notes and findings
* Total wall time is 24s, 7.6s I/O wait. blackfire is likely *significantly* blowing up the total time (my guess would be somewhere around 3-5 times, I can compare execution time without it enabled), but some things like I/O wait shouldn't change too much.
* memory consumption at 200MB, 12MB network transfer (igbinary + compression cuts that down a lot)
* Blackfire itself seems quite confused, it's reporting significantly different numbers in timeline compared to callgraph, no idea why, take all of this with a a lot doubt, especially timings.
* Redis: 1600 cache sets (1.1s in total), 1200 cache reads (800ms). A massive contributor to the cache sets (950) seems to be Drupal\csp\LibraryPolicyBuilder::getLibarySources(), will need to check what that does. csp_rebuild() is 1s. not core, obviously.
* this is not a completely cold cache. FileCache is warm, 2200 getMultiple() calls, no set calls. So no annotation/attribute parsing for plugins and hooks.

Container compilation:
* DrupalKernel::compileContainer() is 7.7s
* HookCollectorPass::collectModuleHookImplementations() is 2s, called 215 times. 80% plus of that time seems to be the directory/file traversing. filterIterator() is called 17923x, reported at 1.5s. Once #3610009: Stop discovery of hooks in include files lands, we can change that to only traverse src/Hook, which should cut this down *a lot*.
* Drupal\Core\DependencyInjection\YamlFileLoader::load(), file cache here only partially warm it seems? 170 calls to load, 65 calls down into loadFile()

Route building

* RouteBuilder::rebuild() total is 13s
* Drupal\views\EventSubscriber\RouteSubscriber::alterRoutes(), calls alterRoutes() on each page display, which all loop over all routes, and call overrideApplies(), that grows exponentialy on number of page displays * number of routes, 88689 calls for me. Cost seems mostly coming from RouteCompiler::getPathWithoutDefaults()
* Drupal\menu_link_content\Plugin\Deriver\MenuLinkContentDeriver
getDerivativeDefinitions is 2.9s, seemingly the source of node loads mentioned below. It finds around 190 menu_link_content entities with the rediscover flag, they all point to a node, and go then through Url::fromInternalUri(), which does Router::match() and canonical upcasting. I've noticed before that I have a lot of those rediscover menu links, don't understand yet exactly what causes them to get that flag.
* MenuTreeStorage::rebuild(), seems to do around 120 saveRecursive() calls and 336 doSave. We have a lot of menus and menu links.

queries:
actual queries are a bit tricky to isolate, because blackfire really doesn't like how we escape our query identifiers and filters out anything as a possible value (queries are reported as SELECT ?.* FROM ? ? WHERE ? IN (:db_condition_placeholder_0)), but I can see where the calls are coming from

* 2800 queries, in total 1.6s actual query time
* 2428 Select query builder queries, 2.3s
* 1000 select queries from NodeStorage::loadFromDedicatedTables() (40%)
* 1000 select queries from MenuTreeStorage::safeExecuteSelect (40%)

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet