Problem/Motivation

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Higher level storage mechanisms like entities and references between them automatically provide cacheability metadata, which is already useful, but AFAICS they do not provide a way for more general queries building non-persisted relationships with cardinality 0-n, meaning they have no automated way to add the existence of that relationship to users of their results. Consider this example, in a headless scenario to put the renderer context out of the way:

* an API response provides a result made of an entity a of type et_a and id id_a, and other entities b* of type et_b related the contents of "a" without any entity_reference relationship between them. Assuming use of actual entities and not just IDs, the result will contain cache tags like: ['et_a:id_a', 'et_b:<id 0>', 'et_b:<id 1>', ...]
* when entity et_b:<id 0> is updated or deleted, an invalidation is emitted for et_b:<id 0>, and the result is therefore invalidated and refreshed, so no problem here
* however when new entities in et_b are created, this result needs to be invalidated too, because new results might be part of the list, but there is no way to relate a new et_b:<id n> entity to that list
* the typical solution for this will be to manually add a tag like et_b_list to the API result (see e.g. ForumBlockBase::getCacheTags()), because every et_b CRUD operation will invalidate et_b_list
* a better scenario would be the ability to add the query instance itself as a cacheable dependency to the result. In initial, basic cases, the cache tags would probably just be ['<entity_type>_list'] based on the fact that entity queries are based on an entity type.

This would allow result building code to apply a more consistent mechanism, adding the query as a cacheable dependency, without having to explicitly know the format of its tags.

In the longer term, it would also pave the path to more sophisticated tags, because they could be generated automatically and become more opaque, removing the temptation to craft them by hand at the application level.

Comments

fgm created an issue. See original summary.

fgm’s picture

Issue tags: +D8 cacheability, +Performance
Berdir’s picture

Interesting idea, but how would they get from the query object to the resulting render array, that would still need to be done manually? The only other option is doing a drupal_render() and that's tricky because entity queries can absolutely happen outside of a render context.

Also, anyone can already define their own more specific cache tags right now, forcing the list cache tag would make that more complicated.

fgm’s picture

In the specific case where we met this need, this is a vanilla JSON API controller (not related with JSONAPI), with no rendering involved.

In such a case, it would be something like:

// In some controller.
$ids = $this->query->execute();
$data = $this->workOn($ids);

$response = new CacheableJsonResponse($data);
$response->addCacheableDependency($this->query);
return $response;

In a render array, it would probably be similar, like:

$ids = $this->query->execute();
$data = $this->workOn($ids);
$build['data'] = $data;
$this->renderer->addCacheableDependency($build['data'], $this->query);
return $build;

The whole idea is to replace the hand-crafted cache tag by one which would automatically come from the query, to make it more abstract and replaceable than the current situation where we manually add cache tag strings. It would not prevent such manual adds, obviously, but remove one need for them.

Wim Leers’s picture

Component: cache system » entity system
Issue summary: View changes
Issue tags: +DX (Developer Experience), +Security improvements

I think the sentiment/intent is right here, but the IS is mixing up lots of things.

+1 for making \Drupal\Core\Config\Entity\Query\Query implement CacheableDependencyInterface, or have its return value implement CacheableDependencyInterface.

Closest related issue: #2557815: Automatically bubble the "user.node_grants:$op" cache context in node_query_node_access_alter().

We never got around to automatically bubbling cacheability metadata for entity queries. We shouldn't actually bubble it, we should be returning a value object that implements CacheableDependencyInterface, so that it's sanely unit-testable. But whether that's still possible without breaking BC is the question.

Note that we did implement this for Views, see \Drupal\views\Plugin\views\cache\CachePluginBase::getCacheTags().


* an API response provides a result made of an entity a of type et_a and id id_a, and other entities b* of type et_b related the contents of "a" without any entity_reference relationship between them. Assuming use of actual entities and not just IDs, the result will contain cache tags like: ['et_a:id_a', 'et_b:<id 0>', 'et_b:<id 1>', ...]

How would the response contain results of multiple entity types? \Drupal\Core\Config\Entity\Query\Query does not support this; it can only operate on a single entity type at a time. So this would require multiple Entity Queries.

* however when new entities in et_b are created, this result needs to be invalidated too, because new results might be part of the list, but there is no way to relate a new et_b:<id n> entity to that list

Why? You first said that it's including entities b* that are related without any entity reference relationship. How can this suddenly/automagically be adding more related entities? Whenever you do an entity query on entity type X, you need to add $this->entityManager()->getDefinition('X')->getListCacheTags() and $this->entityManager()->getDefinition('X')->getListCacheContexts() to the cacheability metadata of the resulting data. That's already a requirement. This issue is just about automating that. (Which would be a huge DX and security win!)

* the typical solution for this will be to manually add a tag like et_b_list to the API result (see e.g. ForumBlockBase::getCacheTags()), because every et_b CRUD operation will invalidate et_b_list

No, ForumBlockBase is a special case, because \Drupal\forum\Plugin\Block\ForumBlockBase::buildForumQuery() uses a "plain" database query, not an Entity Query!

* a better scenario would be the ability to add the query instance itself as a cacheable dependency to the result. In initial, basic cases, the cache tags would probably just be ['<entity_type>_list'] based on the fact that entity queries are based on an entity type.

+1 for this principle!


In short: yes, I'd love to see this happen!

fgm’s picture

Thanks for looking into it. It's indeed a bit confused because the idea happened while working on something else and I wrote it because it seemed important, to avoid forgetting it. By all means feel free to update the IS to be clearer and more targeted :-)

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

geek-merlin’s picture

Title: Implement CacheableDependencyInterface on Entity Query » No sane way to pass Access Cacheability of EntityQuery
Category: Feature request » Bug report

I came across this when i wanted to pass on EntityQuery Access Cacheability.

Looking into #3052553: Entity query alter with cacheable metadata leaks and triggers LogicException, this is how cacheability is passed there:

+  // Bubble the 'user.node_grants:$op' cache context to the current render
+  // context.
+  $renderer = \Drupal::service('renderer');
+  if ($renderer->hasRenderContext()) {
+    $build = ['#cache' => ['contexts' => ['user.node_grants:' . $op]]];
+    $renderer->render($build);
+  }

Which means the hook_entity_access uses the renderer as a global!
Independently if the result is used or not. And what if we store the result in an index and need its cachebility later? Retagging as bug though.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

larowlan’s picture

Title: No sane way to pass Access Cacheability of EntityQuery » Add a way to pass Access Cacheability of EntityQuery
Category: Bug report » Task
Issue tags: +Bug Smash Initiative

Triaged as a daily issue for Bug smash, I don't think this is a bug, its more of a feature, but settled on the middle ground of a task

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.