Problem/Motivation
With the drush migrate-status command in migrate_plus, several migrations defined by migrate_drupal fail when obtaining the source count:
PHP Fatal error: Call to a member function countQuery() on a non-object in /Users/mryan/Sites/d8/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php on line 203
SqlBase obtains the source count via $this->query()->countQuery(), but several migrations in migrate_drupal implement query() thusly:
public function query() {
// Nothing needed here.
}
So, if these migrations have no SQL query, why are they extending DrupalSqlBase rather than SourcePluginBase? If there is a good reason for this, then they should follow the model of CommentVariable, which overrides count() to provide a meaningful value in the absence of a usable SQL query.
Proposed resolution
Fix these migrations, either by extending SourcePluginBase or by providing a count() override.
Remaining tasks
- Submit failing test.
- Fix the migratons.
User interface changes
N/A
API changes
None.
Comment | File | Size | Author |
---|---|---|---|
#29 | interdiff.txt | 418 bytes | mikeryan |
#29 | several_migrate_drupal-2499793-29.patch | 8.79 KB | mikeryan |
Comments
Comment #1
mikeryanPretty simple, actually, to add source count checking for all migrations based on MigrateSqlSourceTestCase. Unfortunately, the fake DB driver doesn't actually catch these failures (although it adds a couple unrelated ones) - @phenaproxima's upcoming patch to replace the fake driver with SQLite should address that.
Comment #2
phenaproximaThe aforementioned SQLite patch may be found at #2499835: Remove broken Fake DB driver.
Comment #3
mikeryanSee #2499835: Remove broken Fake DB driver - with phenaproxima's patch, my fail test still doesn't fail, apparently because the particular problematic source plugins are either untested or using MigrateDrupal6TestBase instead of MigrateSqlSourceTestCase, thus evading the failure... I'll need to add/modify those tests.
Comment #7
mikeryanThe current failures reported have nothing to do with the present issue, fwiw - just another demonstration of why #2499835: Remove broken Fake DB driver is necessary.
Comment #8
mikeryanFake driver is dead, let's see if the fail test can get the right failures now...
Comment #11
mikeryanSo, the expected behavior was that tests for migrations improperly implementing count() would fail and the rest would succeed. What actually happens is that when I run the tests locally they all pass (e.g., UserTest has both an expected and actual count of 2), but testbot failed on all of them. I understand the problem locally - the migrations I'm trying to catch are not based on MigrateSqlSourceTestCase, where I added testSourceCount(), so they're evading my test, and I need to figure out how to properly test them. I have no clue why testSourceCount() fails on everything under the testbot, however. If I force a failure locally I see a message like "Failed asserting that 3 matches expected '2'.", but unfortunately testbot does not provide that detail, which makes it difficult to see what the problem is.
Anyway, in local testing, a partial (but probably not complete) list of problematic migrations includes:
Comment #12
mikeryanOK, this should fail as expected for the broken source plugins. Note that the actual assertions I added aren't getting the chance to fail, because it fatal errors out from under them...
Next - fix it!
Comment #13
mikeryanComment #15
mikeryanOK, here's the patch with the fixes. Note that most of the volume here is simply changing migrate_drupal tests involving a single migration, or a set of migrations with a clear primary migration, to save the migration in the class so the general testSourcePlugin() can test it.
Comment #16
phenaproximaLooks good, except for...
This is repeated several times. Can it be a generic method on the base SQL source?
Comment #17
mikeryanSure... First, a question I meant to ask earlier - any thoughts on the query itself? What's the simplest query to use just for the sake of returning a query object to fulfill the method's contract, in instances where that query will never actually be executed?
Comment #18
phenaproximaPrecisely what's in #16, only with a uniqid() table name (since a dummy query should fail if executed anyway). Contrary to what I said, it should probably be in a trait and NOT in any of the base plugins, since it's a special case that won't apply to most plugins.
Comment #19
mikeryanBefore I submit the updated patch, what's a good name for the generic method (to be called by any source plugins not needing a real query)? dummyQuery()?
Comment #20
phenaproximaThis is blocking #2522012: Remove broken idlist handling, replace with more robust exception handling and #2485385: Move highwater field support to the source plugin, and do not expose its internals on MigrationInterface.
Comment #21
mikeryanHere we are with the trait suggested by phenaproxima. Note the interdiff is sort-of-not-really-incomplete, the patch in #15 no longer applies to MigrateDrupal6TestBase.php, but there were no changes to the changes (if you know what I mean) in that file.
Comment #22
phenaproximaThis patch pleases me.
EDIT: To be clear -- this patch is a kludge. It's critical because it's blocking actual, long-needed, important (as in API-breaking) bug fixes in SqlBase. Having a trait which returns broken, not-to-be-executed dummy queries is weird, but if it allows the bug fixes to move forward, I think it's worth it. SqlBase should be refactored afterwards to make DummyQueryTrait unnecessary; that should probably be in a follow-up issue.
Comment #24
phenaproximaComment #25
mikeryanHere's the reroll, necessitated by #2514168: Streamline migrate_drupal integration tests. I grumbled at all the test changes with merge failures, until I realized that the streamline patch meant I didn't have to change all those tests... interdiff doesn't work because the old patch doesn't apply any more, but this reduces the patch to about 10% its former size.
Comment #27
mikeryanFinally putting that uniqid() into the dummy query exposed a missing count() method...
Comment #28
phenaproximaIt's awesome that you were able to cut this one down to size. Cheers to that :) Back to RTBC!
Comment #29
mikeryanTiny docblock improvement, leaving RTBC with @phenaproxima's blessing.
Comment #30
alexpottMigrate is not frozen in beta and this is a major bug. Committed dfefab5 and pushed to 8.0.x. Thanks!