Problem/Motivation

When a new ECK entity type is created via config import (e.g. drush cim), other modules that provide plugins derived from entity types may fail with "plugin does not exist" errors if those plugins are imported later in the same config import batch.

Root cause

EntityUpdateService::applyUpdates() correctly clears the entity type manager and entity field manager caches after installing a new ECK entity type:

$this->entityTypeManager->clearCachedDefinitions();
$this->entityFieldManager->clearCachedFieldDefinitions();

However, many plugin managers derive their plugins dynamically from the list of available entity types (e.g. Search API's ContentEntityDeriver, which creates an entity:* datasource plugin per entity type). These plugin managers maintain their own separate caches. They are not invalidated by clearing the entity type manager alone.

The result: if config for such a plugin manager (e.g. search_api.index.*) is imported after the ECK entity type config in the same batch, the plugin manager's stale cache still doesn't know about the newly created entity type, and the import fails.

Steps to reproduce

Try to import config containing both a new ECK entity type + a search_api index using that new entity type

You will get something like:

$ drush cim -y
[notice] Synchronized configuration: create eck.eck_entity_type.eck_example.
...
[notice] Synchronized configuration: create search_api.index.index_using_eck_example.
..
[notice] Finalizing configuration synchronization.

In ConfigImportCommands.php line 291:
  The import failed due to the following reasons:
    Unexpected error during import with operation create for search_api.index.index_using_eck_example: The "entity:eck_example" plugin does not exist. Valid plugin IDs for Drupal\search_api\Datasource\DatasourcePluginManager are: ...

In ConfigImportCommands.php line 278:
  Errors occurred during import

Proposed resolution

After clearing entity type caches, also clear all plugin manager caches:
\Drupal::service('plugin.cache_clearer')->clearCachedDefinitions();

In order to clear all derived plugin caches across all plugin managers.

Remaining tasks

?

User interface changes

None

API changes

None

Data model changes

None

CommentFileSizeAuthor
#4 3579703-4.diff2.94 KBherved

Issue fork eck-3579703

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

herved created an issue. See original summary.

herved’s picture

herved commented
Issue summary: View changes

herved opened merge request !72

herved’s picture

herved commented
StatusFileSize
new 3579703-4.diff2.94 KB

Static patch for composer

herved’s picture

herved commented
Status: Active » Needs review

I resisted to use promoted property promotion, delete the constructor docblock and use autowire, as that seemed out of scope

herved’s picture

herved commented
Title: ECK entity type creation during config import does not clear derived plugin caches » Clear all derived plugin caches when an ECK entity type is created/updated
dieterholvoet’s picture

dieterholvoet commented

Clearing all plugin caches seems wasteful and in most cases unnecessary. This sounds like a search_api bug: we're already clearing the entity type definitions, search_api should make sure that ContentEntityDeriver plugins are automatically rebuilt after entity type definitions are rebuilt.

dieterholvoet’s picture

dieterholvoet commented
Related issues: +#3001284: Allow plugin derivers to specify cache tags for their definitions, +#3579730: Clear ContentEntityDeriver plugin caches when entity type definitions are rebuilt

Apparently plugin derivers like ContentEntityDeriver are not yet able to define cache tags that would make it so their definitions are automatically rebuilt at the right time: #3001284: Allow plugin derivers to specify cache tags for their definitions.

I logged this issue in the search_api queue and suggested a workaround: #3579730: Clear ContentEntityDeriver plugin caches when entity type definitions are rebuilt. Could you test if the MR over there also fixes the issue, without the MR here?

dieterholvoet’s picture

dieterholvoet commented
Status: Needs review » Postponed (maintainer needs more info)
herved’s picture

herved commented

Thanks for the feedback.
You're right that conceptually search_api should handle this, but there's no mechanism for a plugin manager to automatically clear its in-memory $this->definitions when another plugin manager does.

I tried the changes to \Drupal\search_api\Datasource\DatasourcePluginManager you proposed in #3579730
But that doesn't work because cache tag invalidation only invalidates the persistent cache backend, it doesn't touch $this->definitions, which is checked first and bypasses the cache backend entirely. So within a single request (which is what config import is), the stale in-memory state persists regardless.

sapnil_biswas made their first commit to this issue’s fork.

herved’s picture

herved commented

Thanks for the tests @sapnil_biswas
But I'm not sure such unit tests are so useful, a functional one that reproduces the bug might be a better choice here.

dieterholvoet’s picture

dieterholvoet commented
Related issues: +#3579767: Static plugin cache is not automatically invalidated

You're right that conceptually search_api should handle this, but there's no mechanism for a plugin manager to automatically clear its in-memory $this->definitions when another plugin manager does.

I logged that as a core issue as well: #3579767: Static plugin cache is not automatically invalidated. Since it doesn't seem like we'll be able to fix the underlying issues anytime soon, I'm okay with committing the workaround. I checked for other examples of plugin definitions that depend on the list of entity types and I found Drupal\linkit\Plugin\Derivative\EntityMatcherDeriver which might be relevant as well.

dieterholvoet’s picture

dieterholvoet commented
Status: Postponed (maintainer needs more info) » Needs work

Setting to NW to improve the tests. Since it's a workaround, we should also add a comment to the new clearCachedDefinitions line explaining in short why it's there and linking to this issue.

herved’s picture

herved commented
Status: Needs work » Needs review
herved’s picture

herved commented

Thanks for reporting those issues.
I used a kernel test to demonstrate this using local task manager.

Usually new entity types are provided by enabling new modules, which clears plugin caches in \Drupal\Core\Extension\ModuleInstaller::doInstall, so I think this issue is quite specific to ECK here and clearing all plugin caches ourselves make sense. But of course it would be ideal if core would handle this.

  • dieterholvoet committed fe1ddd5f on 2.x authored by herved 
    Issue #3579703 by herved, dieterholvoet: Clear all derived plugin caches...
dieterholvoet’s picture

dieterholvoet commented
Status: Needs review » Fixed

Thanks a lot for the contribution!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

thatguy’s picture

thatguy commented

Hello, I updated the ECK module from 2.1.0-beta2 to 2.1.0 and found a strange bug which might be very limited to how my project is setup. Thought I'll share anyway incase someone else faces the same issue.

After updating the module, I deployed the change to testing environment and tested creating a new ECK entity and was faced with a database error which told me that a database table related to one of the fields in the entity type cannot be found. I checked in the database that the table was really missing.

I manually restored the table with a MySQL command and re-deployed the codebase. Again the table was removed from the database. I then reverted the ECK update, restored the table again and deployed. This time the table was not removed.

I have tried to troubleshoot the patch from this issue what could cause it (it's the only change in 2.1.0) but I don't find anything clear. Could be race-conditions or incomplete cache builds during the deployment but very difficult to prove. Also I cannot replicate the issue in my local DDEV environment, it only happens in the AWS environments.

Setup is AWS with Redis as cache backend. Field configurations for the affected field are:

field.field.contact.contact.field_regulations.yml

uuid: <uuid>
langcode: en
status: true
dependencies:
  config:
    - eck.eck_type.contact.contact
    - field.storage.contact.field_regulations
  module:
    - options
id: contact.contact.field_regulations
field_name: field_regulations
entity_type: contact
bundle: contact
label: Regulations
description: 'I want to receive notifications from the company'
required: false
translatable: false
default_value: {  }
default_value_callback: ''
settings: {  }
field_type: list_string

field.storage.contact.field_regulations.yml

uuid: <uuid>
langcode: en
status: true
dependencies:
  module:
    - eck
    - options
id: contact.field_regulations
field_name: field_regulations
entity_type: contact
type: list_string
settings:
  allowed_values:
    -
      value: email
      label: 'By email'
    -
      value: phone
      label: 'By SMS'
  allowed_values_function: ''
module: options
locked: false
cardinality: -1
translatable: true
indexes: {  }
persist_with_no_fields: false
custom_storage: false

And the table that gets lost is called contact__field_regulations. Let me know if I should provide some more info but if nobody else has this issue it might be just related to my specific project caching style or something.

herved’s picture

herved commented

@thatguy I think the first step would be to try to reproduce this outside your project, with a clean drupal core install. Or at least try to understand what kind of setup/scenario could trigger this.
If you are sure this specific change caused a regression it's indeed concerning. OTOH I don't see how clearing all plugins caches would delete an actual field...

thatguy’s picture

thatguy commented

Yep, I'll try to find time to reproduce this or find the cause for this. Just thought to mention it incase someone else would happen to have the same issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.