Problem/Motivation

Right now, updating or maintaining the migration database fixtures is very, very painful and time-consuming task. After a dev exported an updated database fixture, it's very hard to distinguish which changes we want to commit, and which ones we should revert (manually).

Steps to reproduce

Based on the system module's schema, right now the core D7 DB fixture contains a Drupal 7.44 database. Refresh it to the most recent one (7.80), execute database updates, and then export the updated database with

php core/scripts/db-tools.php dump --database fixture_connection > core/modules/migrate_drupal/tests/fixtures/drupal7.php

(See Generating database fixtures for migration tests).

Proposed resolution

Add a new option for the DbDumpCommand which then generates a set of per-table fixture files.

Remaining tasks

  1. Patch
  2. Doc update
  3. Updating the core fixtures (maybe this should happen in a follow-up?)

User interface changes

Nothing

API changes

New option for DbDumpCommand.

Data model changes

Nothing.

Release notes snippet

TBD.

CommentFileSizeAuthor
#2 core-option_creating_db_fixture_split_per_table-3213633-2.patch9.34 KBhuzooka

Comments

huzooka created an issue. See original summary.

huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Assigned: huzooka » Unassigned
Status: Active » Needs review
StatusFileSize
new core-option_creating_db_fixture_split_per_table-3213633-2.patch9.34 KB

I also added a new test for testing the new option; this test now extends the preexisting DbDumpCommandTest.

huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Assigned: Unassigned » huzooka
Status: Needs review » Needs work
Issue tags: +Needs issue summary update

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

quietone commented
Component: migration system » database update system
Issue tags: -migrate-d6-d7

I played with this today and I think this is a good idea. While at first it seemed overwhelming to have all the individual files, it will help one to select the tables needed when updating the database and avoid the irrelevant ones which will have lots of changes.

+++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
@@ -41,7 +42,8 @@ class DbDumpCommand extends DbCommandBase {
+      ->addOption('split-destination', NULL, InputOption::VALUE_OPTIONAL, 'The destination file if per-table-split dump should be generated.', NULL);

This could use a more intuitive option name. Or at least for me. May 'per-table'? I don't know.

Also, the 'destination file' is a directory not a file.

While this will help migrate I am changing component to the database update system, like the related issue.

xjm’s picture

xjm
she/her
Primary language English
commented
Related issues: +#3082230: [meta] Convert some tests to Build tests

 

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

papagrande’s picture

papagrande
Primary language English
commented
Title: Improve DX of mainaining migration database fixtures: provide an option for creating per-table database fixtures in DbDumpcommand » Improve DX of maintaining migration database fixtures: provide an option for creating per-table database fixtures in DbDumpcommand
damienmckenna’s picture

damienmckenna
Location TN, USA
commented

Should I open a separate issue for the problem of certain data structures being stored as serialized arrays with embedded objects ("config" and "key_value" values especially), which makes fixture files extremely complex?

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

quietone commented
Status: Needs work » Needs review

Now that Migrate Drupal is deprecated this isn't needed for that use case. Change to won't fix?

smustgrave’s picture

smustgrave commented
Status: Needs review » Closed (won't fix)

Think so, credit was already saved for the patch attempt FYI.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.