Allow explicit bubbling of cacheability metadata inside Twig template (when accessing data from instead of rendering render arrays)

+++ b/core/lib/Drupal/Core/Template/TwigExtension.php
@@ -342,6 +343,39 @@ public function isUrlGenerationSafe(\Twig_Node $args_node) {
+    $attached_cacheable_metadata = [];
+    $items = [
+      'tags',
+      'contexts',
+      'max-age',
+    ];
+    foreach ($items as $item) {
+      if (!empty($cacheable_metadata[$item])) {
+        $attached_cacheable_metadata['#cache'][$item] = $cacheable_metadata[$item];
+      }
+    }

You can replace that by a single line:

$attached_cacheable_metadata['#cache'] = array_intersect_key($cacheable_metadata, ['tags' => 0, 'contexts' => 1, 'max-age' => 2]);

The logic in templates should only operate on the variables it receives, not on request context (URL, cookies, headers, current user, permissions). As long as that is the case, this is not necessary.

I fear this would complicate our templates too much.

Another classical example is when you use one of the time related twig magics out there. People do it, and well so we need some support for it.

+++ b/core/lib/Drupal/Core/Template/TwigExtension.php
@@ -337,6 +338,27 @@ public function isUrlGenerationSafe(\Twig_Node $args_node) {
+
+    if (empty($attached_cacheable_metadata['#cache'])) {
+      return;
+    }

As talked earlier, we don't care about that case, its will almost never happen.

Curious to know what the alternative to this would be @Wim Leers when you take into consideration people will do what is described in #6?

I see, this is about drillability.

But let's not forget that often the use of drillability means that you'd need to bubble other cacheability metadata. So this would only be a partial solution.

Crosslinking workaround for this issue - https://www.previousnext.com.au/blog/ensuring-drupal-8-block-cache-tags-...

This is a pretty big issue considering that debugging it is not easy to come to the conclusion it is Twig related and what to do about it.

In the case I just came across, the client had blocks they were editing but the blocks were not updating on the node they were edited from. After checking general cache headers, varnish and pin pointing the render_cache I then found that the taginvalidator was correctly invalidating the block and after many more hours I found the workaround to {{ content|without() }} trick and finally got the cachetags to show in the page cacheability metadata (x-Drupal-Cache-Tags).

My point is that many will not easily arrive at Twig being the culprit and the site will just appear broken in that regard and it won't get fixed and the content editors will have a bad Drupal experience. I would consider this priority to be 'major' because of that.

#14 - In my template on a breakpoint there isn't a {{ {'#cache': render_array['#cache']}|render }} only a ['#cache']['contexts'][0] available. I am able to get the cachetags to show up in the page cacheability metadata by doing {{ content|without('field_name_here') }} though, but that is quite tedious with many fields and if more fields are added then that will have to be updated, which would make sense anyways but just pointing out all the issues.

Also, I don't consider this a feature request, it is a bug.

We've been architecting themes heavily based on https://www.drupal.org/project/components. We're running into this issue all over the place as this components-based approach involves a lot of drilling down into render arrays and passing the data we want into component templates as variables. It seems this undermines some of the promise of Twig in terms of drillability (at least how I've perceived it.)

I naively assumed the cache metadata would bubble up just by being present in a render array passed to a template. It makes sense to me now that the template needs to render the render array for the metadata to bubble up. Are there best practices for this documented anywhere? I imagine a lot of themers will shoot themselves in the foot without a better understanding of how drilling into render arrays can bypass the caching layer.

I'm happy to help spread the word, but want to make sure I fully understand the best practices.

Are folks using this patch successfully in production? If not, how do you solve the need?

I was able to get around enumerating fields in my custom block templates by adding {% set catch_cache = content|render %} instead of using the {{ content|without(..) }}. I'm not sure if that has any performance or other negative implications, but I think it worked for me.

pixelwhip said:
>[I] want to make sure I fully understand the best practices.

Do we have any here? At this point I'm moving blocks that need this kind of drill-down access to variables + fine-grain caching to src/Blocks in a module, FWIW, but I'm not sure how good an idea that is or isn't.

froboy: does

{% set catch_cache = content|render %}

work without patching? I'm heading to Try-It-And-See, of course.

1kenthomas: correct. No patch needed.

I started with the article linked above (which is broken up there now) but I was trying to figure out a way to get content to fire all the functions like it's being rendered, but not actually show up in the code anywhere. I tried searching for a twig | with() to try "with none" instead of "without all" but that's not a thing. I thought through the other twig options out there when I remembered seeing something similar on the docs at Twig Field Value. They recommend a similar method to deal with field-level caching:

The field_raw and field_target_entity filters do not provide any cache information. Without additional measures content printed with these filters will not be update when changed in the backend. You can workaround this by rendering the field in your template but not display the result. For example: {% set catch_cache = content.field_image|render %}

I thought I'd try the same thing on the entire content and it seemed to work for me. Adding this line to my block-level templates that don't call {{ content }} and then just clearing caches got all of the right things to bubble. A colleague who was testing my PR ran some Apache Bench tests and reported little performance impact.

So... I'm still not claiming it's great, but it works for us. :)

Just an interesting note.

I found that without the {% set catch_cache = content|render %} line, certain block cache tags do not get properly invalidated, even though they are attached to the page (Seen in the response headers).

I do think this is a bug and having to add the {% set catch_cache = content|render %} to template files should only be a temporary workaround.

With this patch, the cache tags fail to properly invalidate.
At the moment, the {% set catch_cache = content|render %} in template files seems to be the best way.

Jonathan

I was able to get around enumerating fields in my custom block templates by adding {% set catch_cache = content|render %} instead of using the {{ content|without(..) }}.

Does anyone know if using the do tag would also work?

so:
{% do content|render %}

instead of:
{% set catch_cache = content|render %}

I use twigcs as a linter and it complains about un-used variables, so if it works, using do would be a little better.

What's wrong with setting cache metadata like follows? This seems to work for me pretty well.
{{ {'#cache': {contexts: ['url']}} }}

Thanks @chi - just worked for me.

#34 doesn't work when you place a block on a page and then change the contents of the block. The block will never know about the updated content since it doesn't have the proper cache tags.

Rendering the content with |without() works but is error prone. The workaround using {% set catch_cache = content|render %} also works but I can see this having impact on performance since everything will be rendered twice that way.

The block will never know about the updated content since it doesn't have the proper cache tags.

Cache tags can be added the same way.
{{ {'#cache': {tags: ['block_content:123']}} }}

Thank you all for your helpful comments. To sum things up:
We currently don't have a proper solution currently to ensure cache tags bubble up in twig templates, if {{ content }} is not rendered, but only subcontents: {{ content.field_xxx }} which is widely used.

So I also consider this a bug or at least problematic for theme developers.

Available workarounds are:

• {% set catch_cache = content|render %} (seems best but not really nice workaround and the theme developer has to know about this issue (which won't be the case in most situations, so I also consider this a logical bug!) #33 also shows a linting issue
• {{ content|without('list up ALL fields here') }} (seems dirty because if a new field is added you have to expliclitly list it)
• {{ {'#cache': {tags: ['block_content:123']}} }} which requires the theme developer to have deeper knowledge of caching logic which shouldn't be the case!

Further information can be found in this blog article: https://www.previousnext.com.au/blog/ensuring-drupal-8-block-cache-tags-...

As a reply to Win Leers in: #5:

The logic in templates should only operate on the variables it receives, not on request context (URL, cookies, headers, current user, permissions). As long as that is the case, this is not necessary.

The problem exists even when only using field data from content. which is widely used (like the blog entry shows).

Proposed solutions:

So from my perspective (but I have to admit, that I don't know whats technically possible and make sense) we might
a) inherit and render cache tags for single contents in {{ content.XX }} from their parent
b) Add a twig function like {{ content|only('field_name') }} to render the content field without everything but also the parent cache tag (name might be different like {{ content|with() }}) or {{ content|withoutAll }}, ...
c) Change {{ content|without() }} (with no arguments) to render nothing but the cache tags which seems a bit dirty and might break things. But {{ content|without() }} doesn't make much logical sense currently.
d) ?

If someone from the core team agrees or has thoughts, please feel free to copy things over to the issue description.

We currently don't have a proper solution currently to ensure cache tags bubble up in twig templates, if {{ content }} is not rendered, but only subcontents: {{ content.field_xxx }} which is widely used.

The problem exists even when only using field data from content. which is widely used (like the blog entry shows).

Bubbling cache metadata when printing content items i.e. {{ content.field_xxx }} works perfectly for me. If it does not work for you, please submit a bug report with details. This issue is about providing cache metadata in Twig templates when render arrays are not used at all.

@Chi: Well no - it doesn't work for me and it's like @lauriii describes so are you sure it works for you if you ONLY render {{ content.field_xxx }} and NOT {{ content }}?

Our concrete example from block--bundle--card.html.twig is:

   {{
    pattern('card', {
      title: label,
      text: content.body,
      image: content.field_bild,
      button_classes: ['button', 'primary'],
      target_url: target_url,
      link_label: link_label|default('read on'|t),
      link_ext: false
    })
  }}

And from the comments above and our results it seemed to me like I was right with my assumptions in #39. But perhaps I'm not?
Yes, we should clarify this first!

{{ content }} doesn't contain #cache by default, it only contains field render arrays, copied in template_preprocess_node() from 'elements'.

The cacheability metadata is either in fields or one level up on the render array that has #theme => 'node', which should be processed automatically by the Renderer and you shouldn't be able to not have that. However, if you have something else, like another preprocess function that adds cacheability to $variables['content'] then yes, that might get lost.

What information specifically is missing in your example?

In preprocess, you can set $variables['#cache'] directly, ThemeManager supports and processes that, always.

If {{ content.field_xxx }} is rendered without rendering {{ content }}, cacheable metadata from {{ content['#cache'] }} is not bubbled up because the cacheable metadata is bubbled up on render time.

Are we talking about content variable that is part of node.html.twig template? If so how did you find {{ content['#cache'] }}?
On my fresh Drupal 8 installation it is missing.

 array:4 [▼
  "field_image" => array:18 [▶]
  "body" => array:2 [▶]
  "field_tags" => array:2 [▶]
  "links" => array:2 [▶]
]

In our case we experienced the problem in a block_custom template: block--bundle--card.html.twig
where we use pattern() via https://www.drupal.org/project/ui_patterns
We're not changing cache tags.

I'm really sorry if my assumptions and sum-up of the issue were wrong. Maybe this issue shows the same fix (which works for us), but the true reason is the combination with ui_patterns and not rendering (any part of) content directly in the block template file. I'm not sure and will have a closer look into the arrays. Code will speak the truth. Sorry!

@Anybody you may create an issue for ui_patterns project.

Just to provide final information for others who run into a similar issue like described in the blog post and my comments...
Attached you can find the {{ kint() }} dump from within block--bundle--card.html.twig (See #42 for details about the twig contents).

It may or may not be part or related to this issue, at least a case like ours needs a solution like proposed in this issue.

So the difference is that your example is actually inside a block template for a block_content entity and the content there is a very different beast compared to a node.

block_content doesn't have its own template, so yes, content is the full render array and despite the same name, has not much in common with content inside a node template-

I would suggest you create a separate issue for that. It's somewhat related to #2704331: Ability to display block_content entities independently, also outside of Blocks, but I think we can improve that by pushing the cacheability metadata up in the block content block plugin to make it less likely to run into such things.

I added a similar feature to Single File Components that also supports passing cacheable objects, which I found useful: #3109362: Make correctly caching components easier. Also see https://git.drupalcode.org/project/sfc#component-caching for some documentation.

We're currently using the {% set catch_cache = content|render %} trick a lot at work in our Drupal templates that reference non-Drupal Twig files coming from Pattern Lab. We've had some problems with the double rendering causing form errors to be displayed twice when a a Webform inside a Paragraph being rendered by our 'Page section' Twig file (which uses the {% set catch_cache = content|render %} trick) is submitted with form errors.

@samuel.mortenson's approach from his SFC module looks interesting. Am I right in thinking that the cache function linked to in the Gitlab link below would avoid the double rendering as rather than render a render array passed to it, it uses CacheableMetadata::createFromObject to extract just the cache metadata and then returns a render array containing just the metadata?

https://git.drupalcode.org/project/sfc/-/blob/8.x-1.x/src/TwigExtension....

Just want tot chime in to note that #attached libraries are also prone to get lost, which can result in missing styling/js functionality.

Is there some update on recommended approach?

it uses CacheableMetadata::createFromObject to extract just the cache metadata and then returns a render array containing just the metadata?

Yes, it looks like this is what it does.

Drupal 9.4.0 is close so we have a limited time to add any improvements for this problem.

Correct me if I'm wrong, but wouldn't this also be a viable, albeit ugly workaround for folks looking for something more light-weight?

{% extends 'block.html.twig' %}
{% block content %}
  {# manage cacheability #}
  {{ { '#cache': content['#cache'] } }}

  {# Call through to the design system #}
  {% include '@design-system/thing.twig' with content only %}
{% endblock %}

@Luke.Leber that approach works well. The other possible appoach is using cache_metadata filter from Twig Tweak module.

It must be noted that when the content is rendered lazily some cache metadata may not bubble up. So that rendering content is only 100% way to get all cache metata.

-- Edit: My problem seems to be related to Views in combination with Search API and possibly Facets. So this probably doesn't affect all sites --

Any news about this issue? I just came across a website we've update to Drupal 10.1.8 where both workarounds from this issue are not working anymore:

Workaround I always used:

  {% set _render_cache = content|render %}

Alternative workaround:

  {{ { '#cache': content['#cache'] } }}

The provided patches to add the twig function don't apply anymore.

It seems the only way to render the proper cache tags is now to render the entire content variable.