Problem/Motivation
Possible alternative to #3518668: Use Fibers for rendering views rows and related to #1237636: Lazy load multiple entities at a time using fibers.
I think we might be able to lazy/multiple load some more entities if we lazy builder/placeholder entity rendering.
This would have other benefits like reducing cache tags on higher level cache items etc. too. If it does what I think it will do, it could also mean we can remove the 'prepare view' step from entity reference fields, which would be a significant simplification there.
Steps to reproduce
Proposed resolution
Remaining tasks
User interface changes
Introduced terminology
API changes
Data model changes
Release notes snippet
Issue fork drupal-3548293
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
catchComment #3
berdirSimilar concern as in the views issue, special cases such as paragraphs that don't use render caching. For a lazy builder, we could probably combine it with render-cache enabled check, although that couldn't account for cases that remove the cache keys later, such as \Drupal\node\Controller\NodePreviewController::view. On the other hand, paragraphs could possibly benefit a lot from this, as it's common to render a lot of them, with many references to medias, other entities and so on.
I'm still getting familiar with fibers. I did a quick test to just "fiber-ize" EntityViewBuilder::build(), essentially the call there to buildMultiple(). It kind of works, but it didn't quite have the effect that I expected. I think I'm starting to better understand that for fibers to be effective, you need to have multiple of them and loop over them. I thought we can possibly build up some fibers from different entities being rendered as we go, but I think it's not quite so easy.
Somewhat related: I think we should consider to deprecate buildMultiple() and the ability to build multiple entities at once. It's broken (#2843565: getViewBuilder('node')->viewMultiple() bypasses render cache), barely used (comments pretty much being the only case in core AFAIK and they rely on it being broken). Instead, we'd shift towards fibers and stuff to build more entities in parallel.
Comment #4
catchYes this is the reason for trying to use placeholders here rather than adding another fiber loop. With placeholders, we get a 'flat' list of placeholders, which the renderer can then render all together in fibers in the existing placeholder fiber loop.
Nested placeholders (a placeholder that returns another placeholder) then get fibered together a level down etc. etc.
Where we add new fiber loop we can only render the things we have access to in that specific loop. So for the views issue all entities rendered in the rows of one view, or if we did it for entity view multiple, only the entities in that specific call. Which still might be useful but won't catch as many cases.
I did open #3496835: [PP-2] Render children in fibers which results in a lot of 'things at the same level being rendered in fibers' but that would also cover a lot of things that don't need to be.
Comment #5
berdirI thought a bit about how this would work and it's not trivial I think, IMHO, changing what view() returns is problematic, there are going to be so many things that alter and customize that. We technically don't have BC for that, but it's a pretty central piece of what we do and something that's often customized.
For example, something like comment_preview() would be an exception, because you can't add that to a lazy builder. Things like node/comment views Rss row plugins would break too.
We'd also need the ability to opt out for entity types, block_content is already in a lazy builder.
Comment #6
fabianx commentedI think the first step is to check what breaks.
And then deal with the BC problem. (I am sure we can find a solution)
But it would be good to first understand of what would be breaking.
Comment #7
berdirI'm been thinking about this a fair bit and still uncertain on how to approach this exactly and advantages and disadvantages. Implications on big pipe, how pages are rendered and cached (such as a view with 50 teasers).
> I think the first step is to check what breaks.
The thing is that we wouldn't necessarily see what breaks exactly with just core, but I can imagine all kinds of BC issues. Changed rendered arrays that are altered and extended in preprocess as mentioned, but also the _referringItem feature of entity reference formatters for example. It's complex but when correctly used, very useful to influence entities based on where they are rendered.
So far, the best idea I have is to make it an opt-in per entity type, or maybe even a different formatter or formatter setting. I'm struggling to find a way on how it could work with fibers, maybe combined with #3496835: [PP-2] Render children in fibers combined with an opt-in flag, like #render_children_with_fibers = TRUE. But that will presumably only work on a single list of render items.
> If it does what I think it will do, it could also mean we can remove the 'prepare view' step from entity reference fields, which would be a significant simplification there.
I know it's the kind of the opposite of what you'd like but I'm currently considering to instead experiment with expanding on an explicit prepare view step (which might actually just be view, see note below). Essentially, I think it's possible to recursively traverse references on entities considering their view mode setting and configured components for that and preload all entities that will be rendered in a few steps. That currently feels far less daunting, with fewer side effects than fully relying on placeholders/fibers. This should work well for something like a page with lots of paragraphs which render nodes, medias, terms, ...
Based on my custom performance test, I'm currently loading 14 media entities one by one on my example page. Since each has a bunch of fields and also loads the respective file, that's a total of about 200 queries. If I can get them all then my example page is down form 800 queries in 11.2, to around 400 on a cold cache, so a 50% improvement. And at least half of those are config queries and other things that are only needed on really cold caches and following pages will see even bigger relative improvements.
(FWIW, prepareView is already now barely needed since almost nothing in core except comment rendering actually uses buildMultiple(), we render entities one-by-one. We could drop that in favor of just using view() with almost no regression, but deprecating it sounds pretty complicated, as just no longer calling it could would drop optimizations from things relying on it. I just fixed/improved the implemention in entity_reference_revisions but didn't bother making it deal with multiple entities.)