Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.This issue is part of #2157455: [Meta] Make Drupal 8 work with PostgreSQL or remove support from core before release.
Currently there is one or more fail or exception in the migrate_drupal test group identified by the new docker based testbot that need to be fixed to make PostgreSQL finally passing all tests.
Remainig tasks:
- #2477853: PostgreSQL: Add support for reserved field/column names
- #2494415: PostgreSQL: Fix migrate_drupal\Tests\d6\MigrateTaxonomyTermTest
Done:
| Comment | File | Size | Author |
|---|---|---|---|
| #13 | migrate_drupal_results.txt | 224.55 KB | bzrudi71 |
Comments
Comment #1
bzrudi71 CreditAttribution: bzrudi71 commentedMany fails because of schema not found exceptions. May be fixed by #1060476: Multiple issues when PostgreSQL is used with non-public schema, so adding as related.
Comment #2
mradcliffeHere's a patch that will expose test failures in Migrate Drupal by getting rid of the first exception. This prevents the schema generated by Drupal 6 from not working in PostgreSQL because of Drupal Schema types like "text" having explicit "not null" and "length" keys on table definitions. I think this patch needs work. I couldn't find any specification page listing supported types for not null or a way of deriving that information that is scalable.
This brings the next issue, which is something like the following:
Comment #3
BerdirThe map table sounds like the table auto-create problem again, they are dynamically created?
Comment #4
bzrudi71 CreditAttribution: bzrudi71 commentedShould we create sub-issues here, guess we can't handle all that within a single patch? BTW: SQLite doesn't do any better, please see SQLite Bot ;-)
Comment #5
amateescu CreditAttribution: amateescu commentedI just opened a sister issue for SQLite: #2454669: SQLite: Fix tests in migrate_drupal test group
Comment #6
bzrudi71 CreditAttribution: bzrudi71 commentedI took some minutes to see how to move forward here. Attached is a quick patch that should expose the remaining fails and exceptions. It seems we just need to fix the "undefined table" problem and most errors should be gone. I included the workaround for SQLite from @amateescu, but that doesn't seem to work. So again, this is just to have an impression of the remaining fails, not sure when I will have the time to look again here (maybe next week) ;-)
Full test run and patch attached.
Comment #7
amateescu CreditAttribution: amateescu commentedCross-posting my comment from #2477251-2: PostgreSQL: Fix Connection::getFullQualifiedTableName() because this issue has more followers.
Can someone please check if the patch over there in combination with the one from #2 above fixes those undefined table exceptions?
Comment #8
bzrudi71 CreditAttribution: bzrudi71 commentedNice, with all the patches around we get very close to to 100% pass :-) The remaining hand full of fails are about NUT NULL constraints errors:
Drupal\Core\Entity\EntityStorageException: SQLSTATE[23502]: Not null violation: 7 ERROR: null value in column "name" violates not-null constraint DETAIL: Failing row contains (4, forums, en, null, null, null, 0, 1429955198, 1).: INSERT INTO simpletest370267taxonomy_term_field_data (tid, vid, langcode, name, description__value, description__format, weight, changed, default_langcode) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8); Array ( ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->save() (line 929 of /opt/local/apache2/htdocs/drupal8/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).Going to create separate issues now to have a better overview what is left to be done.
Comment #9
bzrudi71 CreditAttribution: bzrudi71 commentedIS update
Comment #10
bzrudi71 CreditAttribution: bzrudi71 commentedCreated #2477845: PostgreSQL: Fix broken migrate table creation.
Comment #11
bzrudi71 CreditAttribution: bzrudi71 commentedCreated #2477853: PostgreSQL: Add support for reserved field/column names.
Comment #12
bzrudi71 CreditAttribution: bzrudi71 commentedAnd the hopefully last one ;-) #2477857: PostgreSQL: Fix not null constraint exceptions in migrate_drupal tests
Comment #13
bzrudi71 CreditAttribution: bzrudi71 commentedJust did a fresh bot run with all patches applied with very lovely results :-)
Comment #14
amateescu CreditAttribution: amateescu commentedNice!
Drupal\migrate_drupal\Tests\d6\MigrateDrupal6Testalso (still) fails on SQLite, even though every migrade_drupal test passes individually ://me looks sideways at the funky code in
Drupal\migrate_drupal\Tests\MigrateFullDrupalTestBase.Comment #15
bzrudi71 CreditAttribution: bzrudi71 commentedCool, #2477251: PostgreSQL: Fix Connection::getFullQualifiedTableName() is already committed, moving out of the way :-)
Comment #16
bzrudi71 CreditAttribution: bzrudi71 commented@amateescu regarding the
Drupal\migrate_drupal\Tests\d6\MigrateDrupal6Test, will that be fixed for PG too? I can't find any open SQLite issue for that?Comment #17
amateescu CreditAttribution: amateescu commentedI'm not sure if it can be fixed for both db drivers at the same time, I haven't started to debug it yet. And there's no issue for SQLite currently, it's just documented as a failing test in #2454669-8: SQLite: Fix tests in migrate_drupal test group. I'll cross-post here if/when I have some updates on it :)
Comment #18
bzrudi71 CreditAttribution: bzrudi71 commentedOpened #2477945: PostgreSQL: Fix migrate_drupal\Tests\d6\MigrateUserProfileFieldTest
Comment #19
mradcliffeI'm going to change component to migration system so it's in the migrate queue, and that will hopefully give this more eyes.
Comment #20
BerdirMight be worth to test with/wait for #2258177: Convert migrate_drupal tests to KernelTestBase to land? It's RTBC and confirmed to fix all remaining sqlite related test fails.
Comment #21
bzrudi71 CreditAttribution: bzrudi71 commentedJust did a fresh migrate_drupal test run, now that #2258177: Convert migrate_drupal tests to KernelTestBase is in. Lovely, we can close:
- #2477857: PostgreSQL: Fix not null constraint exceptions in migrate_drupal tests and
- #2477945: PostgreSQL: Fix migrate_drupal\Tests\d6\MigrateUserProfileFieldTest
There is only one remaining fail related to taxonomy terms.
Comment #22
erik.erskine CreditAttribution: erik.erskine as a volunteer commentedI did some investigation into why
Drupal\migrate_drupal\Tests\d6\MigrateTaxonomyTermTestis failing.Not all taxonomy terms are being migrated - the
taxonomy_term_datatable has just 4 rows on PostgreSQL, whereas there are 6 on MySQL. It looks like this is down to the order in which they are processed during the migration. If we try to migrate a term for which the parent term has not yet been migrated, the child term is skipped. The order seems to vary between MySQL and PostgreSQL.To illustrate this, adding
orderBy('tid')to the query inDrupal\migrate_drupal\Plugin\migrate\source\d6\Termmeans that the parent terms are loaded first, and the tests to pass. However that's only true in this particular case, so it isn't a real solution, but may help with diagnosis.Comment #23
bzrudi71 CreditAttribution: bzrudi71 commentedCleanup!
Comment #24
bzrudi71 CreditAttribution: bzrudi71 commentedCreated #2494415: PostgreSQL: Fix migrate_drupal\Tests\d6\MigrateTaxonomyTermTest as per report by @ingaro in #22.
Comment #25
bzrudi71 CreditAttribution: bzrudi71 commentedMarking as fixed as the remaining (two) issues are now handled within #2157455: [Meta] Make Drupal 8 work with PostgreSQL or remove support from core before release to have them in one place.