Problem/Motivation

With the drush migrate-status command in migrate_plus, several migrations defined by migrate_drupal fail when obtaining the source count:

PHP Fatal error: Call to a member function countQuery() on a non-object in /Users/mryan/Sites/d8/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php on line 203

SqlBase obtains the source count via $this->query()->countQuery(), but several migrations in migrate_drupal implement query() thusly:

  public function query() {
    // Nothing needed here.
  }

So, if these migrations have no SQL query, why are they extending DrupalSqlBase rather than SourcePluginBase? If there is a good reason for this, then they should follow the model of CommentVariable, which overrides count() to provide a meaningful value in the absence of a usable SQL query.

Proposed resolution

Fix these migrations, either by extending SourcePluginBase or by providing a count() override.

Remaining tasks

  1. Submit failing test.
  2. Fix the migratons.

User interface changes

N/A

API changes

None.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mikeryan’s picture

Status: Active » Needs review
FileSize
1.83 KB

Pretty simple, actually, to add source count checking for all migrations based on MigrateSqlSourceTestCase. Unfortunately, the fake DB driver doesn't actually catch these failures (although it adds a couple unrelated ones) - @phenaproxima's upcoming patch to replace the fake driver with SQLite should address that.

phenaproxima’s picture

The aforementioned SQLite patch may be found at #2499835: Remove broken Fake DB driver.

mikeryan’s picture

See #2499835: Remove broken Fake DB driver - with phenaproxima's patch, my fail test still doesn't fail, apparently because the particular problematic source plugins are either untested or using MigrateDrupal6TestBase instead of MigrateSqlSourceTestCase, thus evading the failure... I'll need to add/modify those tests.

Status: Needs review » Needs work

The last submitted patch, 1: several_migrate_drupal-2499793-1-FAIL.patch, failed testing.

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 1: several_migrate_drupal-2499793-1-FAIL.patch, failed testing.

mikeryan’s picture

The current failures reported have nothing to do with the present issue, fwiw - just another demonstration of why #2499835: Remove broken Fake DB driver is necessary.

mikeryan’s picture

Status: Needs work » Needs review

Fake driver is dead, let's see if the fail test can get the right failures now...

Status: Needs review » Needs work

The last submitted patch, 1: several_migrate_drupal-2499793-1-FAIL.patch, failed testing.

mikeryan’s picture

So, the expected behavior was that tests for migrations improperly implementing count() would fail and the rest would succeed. What actually happens is that when I run the tests locally they all pass (e.g., UserTest has both an expected and actual count of 2), but testbot failed on all of them. I understand the problem locally - the migrations I'm trying to catch are not based on MigrateSqlSourceTestCase, where I added testSourceCount(), so they're evading my test, and I need to figure out how to properly test them. I have no clue why testSourceCount() fails on everything under the testbot, however. If I force a failure locally I see a message like "Failed asserting that 3 matches expected '2'.", but unfortunately testbot does not provide that detail, which makes it difficult to see what the problem is.

Anyway, in local testing, a partial (but probably not complete) list of problematic migrations includes:

  • d6_user_picture_field_instance
  • d6_user_picture_entity_display
  • d6_user_picture_entity_form_display
  • d6_upload_field_instance
  • d6_upload_entity_display
  • d6_upload_entity_form_display
mikeryan’s picture

Status: Needs work » Needs review
FileSize
80.33 KB

OK, this should fail as expected for the broken source plugins. Note that the actual assertions I added aren't getting the chance to fail, because it fatal errors out from under them...

Next - fix it!

mikeryan’s picture

Status: Needs review » Needs work

The last submitted patch, 12: several_migrate_drupal-2499793-12-FAIL.patch, failed testing.

mikeryan’s picture

Status: Needs work » Needs review
FileSize
82.9 KB

OK, here's the patch with the fixes. Note that most of the volume here is simply changing migrate_drupal tests involving a single migration, or a set of migrations with a clear primary migration, to save the migration in the class so the general testSourcePlugin() can test it.

phenaproxima’s picture

Status: Needs review » Needs work

Looks good, except for...

   public function query() {
-    // Nothing to do here.
+    // Dummy query, unused because we build the iterator by hand.
+    $query = $this->select('system', 's')
+      ->range(0, 1);
+    $query->addExpression('1');
+    return $query;
   }

This is repeated several times. Can it be a generic method on the base SQL source?

mikeryan’s picture

Sure... First, a question I meant to ask earlier - any thoughts on the query itself? What's the simplest query to use just for the sake of returning a query object to fulfill the method's contract, in instances where that query will never actually be executed?

phenaproxima’s picture

Precisely what's in #16, only with a uniqid() table name (since a dummy query should fail if executed anyway). Contrary to what I said, it should probably be in a trait and NOT in any of the base plugins, since it's a special case that won't apply to most plugins.

mikeryan’s picture

Before I submit the updated patch, what's a good name for the generic method (to be called by any source plugins not needing a real query)? dummyQuery()?

phenaproxima’s picture

mikeryan’s picture

Status: Needs work » Needs review
FileSize
84.57 KB
3.7 KB

Here we are with the trait suggested by phenaproxima. Note the interdiff is sort-of-not-really-incomplete, the patch in #15 no longer applies to MigrateDrupal6TestBase.php, but there were no changes to the changes (if you know what I mean) in that file.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

This patch pleases me.

EDIT: To be clear -- this patch is a kludge. It's critical because it's blocking actual, long-needed, important (as in API-breaking) bug fixes in SqlBase. Having a trait which returns broken, not-to-be-executed dummy queries is weird, but if it allows the bug fixes to move forward, I think it's worth it. SqlBase should be refactored afterwards to make DummyQueryTrait unnecessary; that should probably be in a follow-up issue.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 21: several_migrate_drupal-2499793-21.patch, failed testing.

phenaproxima’s picture

Issue tags: +Needs reroll
mikeryan’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
8.58 KB

Here's the reroll, necessitated by #2514168: Streamline migrate_drupal integration tests. I grumbled at all the test changes with merge failures, until I realized that the streamline patch meant I didn't have to change all those tests... interdiff doesn't work because the old patch doesn't apply any more, but this reduces the patch to about 10% its former size.

Status: Needs review » Needs work

The last submitted patch, 25: several_migrate_drupal-2499793-25.patch, failed testing.

mikeryan’s picture

Status: Needs work » Needs review
FileSize
8.76 KB
469 bytes

Finally putting that uniqid() into the dummy query exposed a missing count() method...

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

It's awesome that you were able to cut this one down to size. Cheers to that :) Back to RTBC!

mikeryan’s picture

Tiny docblock improvement, leaving RTBC with @phenaproxima's blessing.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Migrate is not frozen in beta and this is a major bug. Committed dfefab5 and pushed to 8.0.x. Thanks!

  • alexpott committed dfefab5 on 8.0.x
    Issue #2499793 by mikeryan, phenaproxima: Several migrate_drupal...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.