Problem/Motivation

I discovered a very annoying bug in #3219078: Regression: multilingual taxonomy term migrations are failing if user tries to migrate from Drupal 7.81+ and Entity Translation 7.x-1.1 which highlights that core migration tests are testing the migration of a very outdated Drupal 7 instance. While it seems that most of the contributed modules don't got data schema or value updates, settings up an outdated codebase (which works only with unsupported PHP versions) is a constant challenge. ...and it also gives us a false sense of "since core tests are passing, we definitely can migrate most of the Drupal 7 config + data into Drupal9". See #3219078: Regression: multilingual taxonomy term migrations are failing if user tries to migrate from Drupal 7.81+ and Entity Translation 7.x-1.1 .

Steps to reproduce

For an example check #3219078: Regression: multilingual taxonomy term migrations are failing if user tries to migrate from Drupal 7.81+ and Entity Translation 7.x-1.1

Proposed resolution

Update the core database fixture. this means:

  1. Scan the system table for module versions (assuming that it contains the right info).
  2. Build a codebase with the outdated core + contribs. Make sure you use the right PHP version.
  3. Import the DB fixture.
  4. Update the codebase with the most recent core and contribs.
  5. Execute the database updates.
  6. Identify which manual fixture edits caused the errors you see, and try to fix them.
  7. Repeat #3, #4 and #5 until you think that the database integrity is fixed.
  8. Export the updated DB and add that to core.
  9. Preferred: document the versions of core + contrib modules – I would add a drush make file for example.

Imho #3213633: Improve DX of maintaining migration database fixtures: provide an option for creating per-table database fixtures in DbDumpcommand can help us a lot, because it allows you to easily skip adding unnecessary changes made in tables on a per-table-basis, and not only on a per-hunk basis.

Remaining tasks

  • Triage this issue
  • Agree on a solution.
  • Do it.

User interface changes

Nothing.

API changes

Nothing.

Data model changes

Nothing.

Comments

huzooka created an issue. See original summary.

huzooka’s picture

Issue summary: View changes
quietone’s picture

StatusFileSize
new801.49 KB

I installed the fixture to my existing (outdated) Drupal 7 test site, adding anonymous and cleared cache with drush. Then exported and made a patch.

quietone’s picture

StatusFileSize
new936.92 KB

Updated to Drupal 7.82, updated all modules and ran db updates.

quietone’s picture

I need to amend that. I did not update all contrib modules, I only did the ones that are enabled.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

Issue tags: -migrate
StatusFileSize
new887.99 KB

I've started over on this. I have updated to Drupal 7.92 and also the latest versions of all the installed modules. Just like before this db mostly works in a drupal 7 site. One thing to fix, is that viewing any of the article nodes still results in an error.
EntityReferenceRecursiveRenderingException: Recursive rendering detected when rendering entity node(3). Aborting rendering. in entityreference_field_formatter_view() (line 1420 of /var/www/html/drupal/sites/all/modules/entityreference/entityreference.module).

Before addressing those I am running the tests.

quietone’s picture

Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch, 8: 3219539-8.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new2.17 MB

Looks like I forgot to add the Breakpoints and Picture modules. This add those which should fix all the texts except for MigrateNodeCounterTest.

This also fixes the format of Reference2 at admin/structure/types/manage/article/display. I learned from experience that this was prevented using the fixture from the UI on a D7 site.

Status: Needs review » Needs work

The last submitted patch, 11: 3219539-11.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new1.18 MB

The modules on the source site were not in the right state, dashboard needed to be disabled, and syslog and testing needed to be enabled. This will not fix everything :-(

quietone’s picture

StatusFileSize
new1.14 MB

This time disable picture module and breakpoint module. Also made sure to truncate accesslog and to restore the original node counter table data.

The last submitted patch, 13: 3219539-13.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 14: 3219539-15.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new1.14 MB

Looks like I didn't restore the node counter table. Trying again.

quietone’s picture

StatusFileSize
new4.36 KB

Assuming tests pass then the fixture has been updated. I've added a make file.

And for completeness noting other problems with the fixture. There problems existed before the update.

1) /admin/modules - Enabling a module or modules that require the use of the confirm form will cause many modules to be uninstalled.
2) Navigating to /is/firefly-is or /is/deep-space-9-is results in the error shown below. This can be fixed by changing the format of 'Reference 2' from 'Rendered entity' to 'Entity id' at admin/structure/types/manage/article/display. That would require a change to d7/MigrateFieldFormatterSettingsTest.php which is work with little gain. If the node needs to be edited just change the format.

EntityReferenceRecursiveRenderingException: Recursive rendering detected when rendering entity node(4). Aborting rendering. in entityreference_field_formatter_view() (line 1420 of /var/www/html/drupal/sites/all/modules/entityreference/entityreference.module).

heddn’s picture

Status: Needs review » Reviewed & tested by the community

Assuming this comes back green on 10.x and 10.1, this is RTBC.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

It didn't come back green on 10.x

mradcliffe’s picture

A slightly more helpful diff of the fail using s/, /\r/g reveals:

- Administer fields was added in Drupal 7.50 - https://www.drupal.org/node/2483307
- Dashboard and Overlay removed from Drupal 8 - https://www.drupal.org/node/2116417

diff -dub expected.txt actual.txt
--- expected.txt        2022-10-05 17:46:29.679130279 -0400
+++ actual.txt  2022-10-05 17:46:21.927130353 -0400
@@ -2,9 +2,7 @@
 'access comments'
 'access content overview'
 'access contextual links'
-'access dashboard'
 'access news feeds'
-'access overlay'
 'access printer-friendly version'
 'access site-wide contact form'
 'access statistics'
@@ -17,6 +15,7 @@
 'administer comments'
 'administer contact forms'
 'administer content types'
+'administer fields'
 'administer filters'
 'administer forums'
 'administer image styles'
quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new788 bytes
new1.14 MB

I was going to test on D10 this morning but I see it has already happened and the results have been diagnosed. Thanks @heddn and @mradcliffe.

And here is the patch for D10.

heddn’s picture

Status: Needs review » Reviewed & tested by the community

+1 on RTBC from me, given the original scope of things here. Can we improve the fixture or make it less complicated. Assuredly we could. But let's do that in a follow-up.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed 7610d3c1ef to 10.1.x and 7225a0b56e to 10.0.x. Thanks!
Committed abde57f and pushed to 9.5.x. Thanks!

We have better tests on D10 thanks to #2953111: Only migrate role permissions that exist on the destination

  • alexpott committed 7610d3c on 10.1.x
    Issue #3219539 by quietone, mradcliffe: Update Drupal 7 migration...

  • alexpott committed 7225a0b on 10.0.x
    Issue #3219539 by quietone, mradcliffe: Update Drupal 7 migration...

  • alexpott committed abde57f on 9.5.x
    Issue #3219539 by quietone, mradcliffe: Update Drupal 7 migration...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.