Problem/Motivation

In SourcePluginBase, if a cache_key is not specified, one is generated by taking a sha256 hash of the source pluginID.

    if (!isset($this->cacheKey)) {
      $this->cacheKey = hash('sha256', $this->getPluginId());
    }

This is incorrect, as the same source plugin may be used in multiple migrations with different configuration which affect the source count. For example, derived D7 node migrations will use the same d7_node source but with different bundle configurations to retrieve each content type, and each content type will have a different count. Due to #2598670: Source count caching does not work for SQL sources, the cache doesn't come into play for sql sources, which would cover most of core's use case, but if multiple migrations use say the contrib 'url' plugin and enable cache_counts without setting a cache_key, they will all interfere with each other.

Proposed resolution

Set the default cache key to a hash of the entire configuration array, and not just the plugin id.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mikelutz created an issue. See original summary.

mikelutz’s picture

mikelutz’s picture

Status: Needs work » Needs review
FileSize
7.41 KB
1.91 KB

And a possible fix

mikelutz’s picture

Status: Needs review » Postponed

Alright, I guess core_version_requirement: * doesn't work. ^8 || ^9 probably would but I don't want to leave it in there, so lets see if #3072702: Core extensions should not need to specify the new core_version_requirement in *.info.yml files so that 9.0.x can be installed gets committed soon.

The last submitted patch, 6: 3092227-6.TEST_ONLY.drupal.patch, failed testing. View results

heddn’s picture

Status: Needs review » Needs work

No problems here with the implementation. Just some minor nits on on comments.

  1. +++ b/core/modules/migrate/tests/modules/migrate_cache_counts_test/src/Plugin/migrate/source/CacheableEmbeddedDataSource.php
    @@ -0,0 +1,32 @@
    +class CacheableEmbeddedDataSource extends EmbeddedDataSource {
    +
    +  /**
    +   * {@inheritdoc}
    +   */
    +  public function count($refresh = FALSE) {
    +    return SourcePluginBase::count($refresh);
    +  }
    +
    +  /**
    +   * {@inheritdoc}
    +   */
    +  protected function doCount() {
    +    return parent::count(TRUE);
    +  }
    

    I'm a little confused with why we are statically calling SourcePluginBase::count() here. Perhaps an inline comment would clarify.

  2. +++ b/core/modules/migrate/tests/src/Kernel/Plugin/MigrationSourceCacheTest.php
    @@ -0,0 +1,122 @@
    +    // Verify correct counts are cached.
    +    $this->assertSame(1, $migration_1_source->count());
    +    $this->assertSame(2, $migration_2_source->count());
    

    This doesn't really verify caching as best as it could. However, we do have more testing of that later. Maybe just a comment clarification of what is really being tested is in order.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Wim Leers’s picture

mikelutz’s picture

I closed the other issue as a duplicate since this one has work done in it, and the other does not.

Wim Leers’s picture

Marked #2751829: Default source plugin cache key is insufficiently unique as a duplicate of this.

EDIT: cross-posted, hah!

Wim Leers’s picture

+++ b/core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php
@@ -199,7 +200,7 @@ public function __construct(array $configuration, $plugin_id, $plugin_definition
-    $this->cacheKey = !empty($configuration['cache_key']) ? $configuration['cache_key'] : NULL;
+    $this->cacheKey = $configuration['cache_key'] ?? hash('sha256', Json::encode($configuration));
     $this->idMap = $this->migration->getIdMap();
     $this->highWaterProperty = !empty($configuration['high_water_property']) ? $configuration['high_water_property'] : FALSE;
 
@@ -450,10 +451,6 @@ public function count($refresh = FALSE) {
       return -1;
     }
 
-    if (!isset($this->cacheKey)) {
-      $this->cacheKey = hash('sha256', $this->getPluginId());
-    }

This is also going to result in cache collisions, because now the plugin ID is ignored entirely.

It should be cached by both plugin ID and its configuration.

Now … you might say … that in practice, migration definitions have plugin: <source PLUGIN ID>. But that's not actually necessary.

Furthermore, having a cache ID that is literally just a hash is going to be a DX nightmware.

So I propose to keep your logic but prefix it with the source plugin ID 🤓 WDYT?

Wim Leers’s picture

mikelutz’s picture

Prefixing the plugin_id or even using a serialized array with plugin_id and config_hash keys for readability would be fine with me.

Wim Leers’s picture

Cool!

quietone’s picture

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

alexpott’s picture

Version: 9.3.x-dev » 9.1.x-dev
Status: Needs work » Closed (duplicate)

I think this closed now that #3190815: Source count caching broken: impossible to enable source count caching for SqlBase-based source plugins (plus, unneeded cache I/O) has landed. Changing to 9.1.x since that issue landed in that branch.