Problem/Motivation

There is a design pattern that derivers statically cache the resulting derivatives (e.g. \Drupal\Component\Plugin\Derivative\DeriverBase::$derivatives). This can result in out of date derivatives data to be rebuild and cached, despite other things correctly calling clearing the plugin manager caches. The example we have in the Contacts module is:

  1. Install Contacts module
  2. Some stuff happens which involves getting definitions from the block manager (possibly importing page manager pages?). This primes the static caches.
  3. Facet config entities are imported. Facets (with patch in #2880411-6: Facets added via config don't clear the block cache) clear the block plugin cached definitions expecting those blocks now to be found next time the definitions are rebuild.
  4. Some stuff happens which gets the definitions from the block manager (possible other page manager pages?). This results in using the existing static cache and returning the old, now stale, derivatives.
  5. Because the cache has been rebuilt, future requests use this already stale version of the cache.

This can be worked around in contrib by clearing the plugin.manager.block service from the container in addition to clearing the cached definitions.

Proposed resolution

Not sure what should actually be done, initial thoughts on possible solutions:

  • Stop statically caching... If we attempt to get derivatives twice in a request that will be because the cache has been cleared and we need to get them fresh. I don't know if this is just a documentation/change record and then fix in contrib or if there are BC issues (e.g. is \Drupal\Component\Plugin\Derivative\DeriverBase public API or internal)...
  • Don't store the derivers as part of \Drupal\Component\Plugin\Discovery\DerivativeDiscoveryDecorator, which effectively renders the static caches irrelevant. This could have a performance overhead (though again a questionable benefit in the first place?), but fix all contrib even if it continues using the static pattern. Probably worth doing above in addition...
  • Make \Drupal\Core\Plugin\DefaultPluginManager::clearCachedDefinitions clear itself off the container or clear the cached derivers...

Remaining tasks

Decide how to solve this...

User interface changes

None.

API changes

Depends on the decision...

Data model changes

None.

Issue fork drupal-2880682

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

andrewbelcher created an issue. See original summary.

tim.plunkett’s picture

andrewbelcher’s picture

Possibly, I think they may both describe a similar problem, but in this case the issue isn't the module install, it's config installation, so simply resetting at module install won't catch this scenario.

tim.plunkett’s picture

I agree. But I think this will be blocked on that issue, as it adds the necessary $this->discovery = NULL; change.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

larowlan’s picture

Another case #2902281: Configuration import fails

The issue there is \Drupal\Component\Plugin\Discovery\DerivativeDiscoveryDecorator::getDeriver has a static cache

larowlan’s picture

this at least unblocks my project, no tests etc

sticking here for composer patches sake

tim.plunkett’s picture

  1. +++ b/core/lib/Drupal/Core/Plugin/Discovery/ContainerDerivativeDiscoveryDecorator.php
    @@ -32,4 +33,26 @@ protected function getDeriver($base_plugin_id, $base_definition) {
    +   * Disable the use of caches.
    +   *
    +   * Can be used to ensure that uncached plugin definitions are returned,
    +   * without invalidating all cached information.
    +   *
    +   * This will also remove all local/static caches.
    +   *
    +   * @param bool $use_caches
    +   *   FALSE to not use any caches.
    +   */
    

    Can also be {@inheritdoc}

  2. +++ b/core/lib/Drupal/Core/Plugin/Discovery/ContainerDerivativeDiscoveryDecorator.php
    @@ -32,4 +33,26 @@ protected function getDeriver($base_plugin_id, $base_definition) {
    +    unset($this->derivers);
    

    This is a weird method. But AFAICS this unset should be wrapped in a if (!$use_caches) {

larowlan’s picture

Status: Active » Needs work
+++ b/core/lib/Drupal/Core/Plugin/DefaultPluginManager.php
@@ -192,6 +192,9 @@ public function clearCachedDefinitions() {
+    if ($this->discovery instanceof CachedDiscoveryInterface) {

we should trigger a deprecated error in the else section, so we can remove the instanceof check in d9 and force discoveryinterface to extend the cached version

agree on all points from @tim.plunkett review

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

larowlan’s picture

rigoucr’s picture

Status: Needs work » Needs review

I think that @larowlan forgot about to change this issue status from needs work to needs review when sent the re-roll of the patch
Changing issue status to needs new

rigoucr’s picture

Status: Needs review » Reviewed & tested by the community

I was having this issue: https://www.drupal.org/project/search_api_autocomplete/issues/2902281#co... during the installation of my site and I solved it applying the patch in comment #12. It works for me.

alexpott’s picture

+++ b/core/lib/Drupal/Core/Plugin/Discovery/ContainerDerivativeDiscoveryDecorator.php
@@ -32,4 +33,26 @@ protected function getDeriver($base_plugin_id, $base_definition) {
+  public function useCaches($use_caches = FALSE) {
+    unset($this->derivers);
+  }

We're adding a method with no usages - that tends to get us into trouble.

There's also no test coverage.

I suspect that #3001284: Allow plugin derivers to specify cache tags tags for their definitions might also help with this.

mpp’s picture

https://www.drupal.org/project/search_api_autocomplete/issues/2902281

I'm adding the issue mentioned by @rigoucr. The search_api_autocomplete SearchPluginDeriverBase has a static cache $derivatives that breaks our config import:

 [notice] Synchronized configuration: create views.view.contacts.
 [notice] Synchronized configuration: create search_api_autocomplete.search.contacts.
 [notice] Synchronized configuration: create views.view.sg_search.
 [notice] Synchronized configuration: create search_api_autocomplete.search.sg_search.
 [notice] Finalizing configuration synchronization.
In ConfigImportCommands.php line 259:

Unexpected error during import with operation create for search_api_autocomplete.search.sg_search: Unknown search plugin with ID 'views:sg_search'

On the first run of ViewsDeriver::getDerivativeDefinitions the static cache is created (for search_api_autocomplete.search.contacts) but for the second one (search_api_autocomplete.search.sg_search) it uses the static cache that doesn't know about the second view (views.view.sg_search).
The static cache isn't updated when a another view is added (either by module installation or config sync).

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

larowlan’s picture

karishmaamin’s picture

Status: Needs work » Needs review
FileSize
2.05 KB

Re-rolled patch against 9.4.x. Please review

larowlan’s picture

Priority: Normal » Minor
Status: Needs review » Needs work

Back to needs work for #15

immaculatexavier’s picture

Rerolled against #15

immaculatexavier’s picture

Status: Needs work » Needs review

Rerolled against #15

borisson_’s picture

Back to NW, #15 needs to be resolved (tests).

The method added (useCaches) was not implemented in the last patch (#26), which is why it fails

borisson_’s picture

Status: Needs review » Needs work
immaculatexavier’s picture

Status: Needs work » Needs review
FileSize
1.78 KB
503 bytes

Addressed fix against #28 with interdiff

immaculatexavier’s picture

Addressed fix against #30 with interdiff

borisson_’s picture

Issue tags: -Needs reroll

In #15 @alexpott said this, but this is a method that is implemented because the interface needs it. That's also why #26 failed.

We're adding a method with no usages - that tends to get us into trouble.

So all we need now is for the tests to be added, they can be similar to the tests that were needed to test the CachedDiscoveryInterface in other places I think. Or is it just that we want to test that the cached discovery is implemented? I don't think we need to actually test the caching behavior again here?

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

smustgrave’s picture

Status: Needs review » Needs work
Issue tags: +Needs Review Queue Initiative

This issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue [2700+ issues] to around 400 (1 month or less), following Review a patch or merge request as a guide.

Moving to NW for the test cases.

Also IS has several proposed solutions. Can we highlight which was chosen please

Thanks

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

zeeshan_khan made their first commit to this issue’s fork.

zeeshan_khan’s picture

Rerolled patch for 10.1.x, I needed this for very important project.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.