Problem/Motivation

The media_migration plugins are some of the slowest, due to the complexity of their queries. It is crucial that they are allowed to be made cacheable.

See \Drupal\migrate\Plugin\migrate\source\SourcePluginBase::count() + \Drupal\migrate\Plugin\migrate\source\SourcePluginBase::$cacheCounts.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Comments

Wim Leers created an issue. See original summary.

wim leers’s picture

Status: Active » Needs review
StatusFileSize
new1.79 KB
wim leers’s picture

Title: Allow source counts to be made cacheable: implement ::doCount() instead of ::count() » Allow source counts to be cached: implement ::doCount() instead of ::count()

Status: Needs review » Needs work

The last submitted patch, 2: 3190800-2.patch, failed testing. View results

wim leers’s picture

Title: Allow source counts to be cached: implement ::doCount() instead of ::count() » [PP-1] Allow source counts to be cached: implement ::doCount() instead of ::count()
Status: Needs work » Postponed
wim leers’s picture

Issue tags: +Novice
+++ b/src/Plugin/migrate/source/d7/FileEntityConfigSourceBase.php
@@ -164,7 +164,7 @@ abstract class FileEntityConfigSourceBase extends ConfigSourceBase {
-  public function count($refresh = FALSE) {
+  public function doCount() {

Nit: should be protected.

huzooka’s picture

Category: Bug report » Task

This is not a Media Migration bug – from Media Migration's perspective, this is a task.

wim leers’s picture

Title: [PP-1] Allow source counts to be cached: implement ::doCount() instead of ::count() » Allow source counts to be cached: implement ::doCount() instead of ::count()
Status: Postponed » Needs review
damienmckenna’s picture

AFAICS this would break compatibility with D8.9, right?

damienmckenna’s picture

Could we retain the old method and wrap the new one?

huzooka’s picture

@DamienMcKenna, that's the plan.

But imho, don't migrate into 8.9.x :)

huzooka’s picture

Assigned: Unassigned » huzooka
Status: Needs review » Needs work
huzooka’s picture

Assigned: huzooka » Unassigned
Status: Needs work » Needs review
StatusFileSize
new8.58 KB
new8.69 KB
huzooka’s picture

Assigned: Unassigned » huzooka
Status: Needs review » Needs work
huzooka’s picture

Assigned: huzooka » Unassigned
Status: Needs work » Needs review
Issue tags: -Novice
StatusFileSize
new12.27 KB
new3.7 KB
huzooka’s picture

StatusFileSize
new13.14 KB
new6.77 KB
wim leers’s picture

Status: Needs review » Reviewed & tested by the community
  1. +++ b/src/Plugin/migrate/source/d7/ConfigSourceBase.php
    @@ -13,4 +14,26 @@ abstract class ConfigSourceBase extends DrupalSqlBase {
    +   * @todo Remove after Drupal core 8.x and 9.0.x are out of support.
    

    Übernit: s/out of support/unsupported/ 🤓

  2. +++ b/src/Plugin/migrate/source/d7/ConfigSourceBase.php
    @@ -13,4 +14,26 @@ abstract class ConfigSourceBase extends DrupalSqlBase {
    +    if (MigrationPluginTool::sourcePluginCountIsCacheable()) {
    +      return parent::count($refresh);
    +    }
    +
    +    return $this->initializeIterator()->count();
    

    This does: "When cacheable, do not call our override, but instead call the parent implementation, which is what will call ::doCount() on 9.1 & 9.2, but not 8.9 & 9.0." 👍

  3. +++ b/src/Utility/MigrationPluginTool.php
    @@ -11,6 +11,56 @@ use Drupal\Core\Plugin\PluginBase;
    +  /**
    +   * Determines whether SqlBase source plugins might have cacheable counts.
    +   *
    +   * @todo Remove after Drupal core 8.x and 9.0.x are out of support.
    +   *
    +   * @see https://drupal.org/i/3215106
    +   *
    +   * @return bool
    +   *   TRUE if migration source plugins extending SqlBase might have cacheable
    +   *   counts, FALSE otherwise.
    +   */
    +  public static function sourcePluginCountIsCacheable() :bool {
    

    🤓 Nit: I think the name is a bit misleading here.

    I think
    public static function cacheableSourceCountSupportedByCore() would be clearer.

  4. +++ b/src/Utility/MigrationPluginTool.php
    @@ -11,6 +11,56 @@ use Drupal\Core\Plugin\PluginBase;
    +    // Drupal core 9.1 is the oldest minor which got the fix - but every release
    +    // prior to 9.1.9 does not have cacheable source plugin counts.
    +    if (
    +      version_compare($version_normalized, '9.1.9', 'ge') &&
    +      version_compare($version_normalized, '9.2', 'lt')
    +    ) {
    +      return TRUE;
    +    }
    +
    +    // Core 9.2 versions: alpha1 was released without the fix.
    +    if (version_compare($version_normalized, '9.2.0-alpha2', 'ge')) {
    +      return TRUE;
    +    }
    

    👏 This is next-level BC — even includes consideration for alpha releases!

  5. +++ b/tests/src/Kernel/Plugin/migrate/source/d7/MediaMigrationSourceTestBase.php
    @@ -61,11 +62,41 @@ abstract class MediaMigrationSourceTestBase extends MigrateSqlSourceTestBase {
    +    if (MigrationPluginTool::sourcePluginCountIsCacheable()) {
    +      /** @var \Drupal\Core\Cache\MemoryCounterBackend $cache **/
    +      $cache = \Drupal::cache('migrate');
    +      if ($expected_cache_key) {
    +        // Verify the the computed cache key.
    +        $reflector = new \ReflectionObject($plugin);
    +        $property = $reflector->getProperty('database');
    +        $property->setAccessible(TRUE);
    +        $this->assertSame($expected_cache_key, $property->getValue($plugin));
    +
    +        // Cache miss prior to calling ::count().
    +        $this->assertFalse($cache->get($expected_cache_key, 'cache'));
    +
    +        $this->assertSame([], $cache->getCounter('set'));
    +        $count = $plugin->count();
    +        $this->assertSame($expected_count, $count);
    +        $this->assertSame([$expected_cache_key => 1], $cache->getCounter('set'));
    +
    +        // Cache hit afterwards.
    +        $cache_item = $cache->get($expected_cache_key, 'cache');
    +        $this->assertNotSame(FALSE, $cache_item, 'This is not a cache hit.');
    +        $this->assertSame($expected_count, $cache_item->data);
    +      }
    

    This is copied from the core test coverage that I worked on.

    👍 I think this is to be 100% certain that source count caching works as expected — and that's fine of course!

huzooka’s picture

StatusFileSize
new4.19 KB

A new approach, without interdiff.

Status: Reviewed & tested by the community » Needs work
huzooka’s picture

Status: Needs work » Needs review
StatusFileSize
new4.59 KB
new1.01 KB

Plain file config source plugins also need the trait

huzooka’s picture

Assigned: Unassigned » huzooka
Status: Needs review » Needs work
huzooka’s picture

Assigned: huzooka » Unassigned
Status: Needs work » Needs review
StatusFileSize
new5.02 KB
new2.93 KB

Cannot test the trait-based approach with PhpUnit 6/7, let's go back to an abstract class.

huzooka’s picture

StatusFileSize
new5.29 KB
new1.51 KB

Coding standards. Seems that this is not my day 😞

  • huzooka committed 545bd09 on 8.x-1.x
    Issue #3190800 by huzooka, Wim Leers: Allow source counts to be cached:...
huzooka’s picture

Let's see how this works without the DummyQueryTrait::count method aliasing in MediaViewMode.

huzooka’s picture

StatusFileSize
new3.54 KB

#26 was still wrong.

If we are on a Drupal core version which supports cacheable source plugin item counts, then DummyQueryTrait has a doCount() method.

huzooka’s picture

StatusFileSize
new3.54 KB

...aaaaaand: Coding standard!

huzooka’s picture

StatusFileSize
new2.67 KB

#27 is wrong, DummyQueryTrait::doCount comes from one of our core patches.

huzooka’s picture

StatusFileSize
new5.93 KB

...but it's worth supporting that core patch as well.

And imho this is the right time to add a source plugin test for MediaViewModes.

  • huzooka committed 688bbbb on 8.x-1.x
    Issue #3190800 by huzooka, Wim Leers: Allow source counts to be cached:...
huzooka’s picture

Status: Needs review » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.