Currently using migrate tools to work on a rather complex migration. Profiles come in, and some of the data creates paragraphs, and some of those paragraphs go onwards to generate group content. Deletion is just as complex. Remove a node, remove paragraphs, remove paragraphs, remove group content.

With 1059 records (a sample of the full amount we have to migrate) whenever I run migrate currently I get a 504 bad gateway response. The rollback runs partially through the records to remove, but ultimately times out. Updating this to using a batch process would be helpful.

CommentFileSizeAuthor
#12 migrate_tools-batch-rollback-reroll-for-6_0_x-3014353-12.patch7.29 KBygstnd
#11 migrate_tools-batch-rollback-reroll-for-6_0_x-3014353-11.patch7.29 KBovanes
#3 migrate_tools-batch_rollback-3014353-3.patch8.73 KBvasike
#3 migrate_tools-6.0.x-batch_rollback-3014353-3.patch8.4 KBvasike
#2 migrate_tools-batch_rollback-3014353-2.patch8.1 KBkiseleva.t

Issue fork migrate_tools-3014353

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

jnicola created an issue. See original summary.

kiseleva.t’s picture

kiseleva.t commented
Version: 8.x-4.0 » 8.x-5.0
Status: Active » Needs review
StatusFileSize
new migrate_tools-batch_rollback-3014353-2.patch8.1 KB

Implemented batch for rollback (for version 8.x-5.0), for now, made it configurable by a checkbox in "Additional execution options".

vasike’s picture

vasike
Location Ramnicu Valcea
commented
Version: 8.x-5.0 » 8.x-5.x-dev
StatusFileSize
new migrate_tools-6.0.x-batch_rollback-3014353-3.patch8.4 KB
new migrate_tools-batch_rollback-3014353-3.patch8.73 KB

There 2 re-rolls for this ... one for 5.x and the other for 6.0.x

it's @kiseleva.t code ... nothing new added by me.

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented
Version: 8.x-5.x-dev » 6.0.x-dev

Updating to latest version, where development mainly happens. #3 already contains a patch against that version.

acidaniel’s picture

acidaniel commented

It is possible to use drush to pass any `--batch_size` option, I can't see this is something defined here Incorporating that option will be AMAZING!

heddn’s picture

heddn
Primary language English
Location Illinois
commented
Status: Needs review » Needs work

Can we just always use a batch for rollback? I can't think of a reason to use the old way when batch is better.

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

Totally agree with #6!

@kiseleva.t was there a technical reason for the checkbox / option? Or just to let the user decide and make the switch more soft?

vasike opened merge request !39

vasike’s picture

vasike
Location Ramnicu Valcea
commented
Status: Needs work » Needs review

i created a MR from the previous patches ... replacing the rollback with batch rollback.

Maybe some code could be cleanup ... so anyone that wants to. help, please update the MR ...

jaypan’s picture

jaypan
Location Victoria, BC
commented
Priority: Normal » Major

I've confirmed this is working, and it is actually essential for me as my system times out and rollbacks are impossible when they are not done through a batch. I'm marking this as major, as this bug prevents module functionality for me.

ovanes’s picture

ovanes commented
StatusFileSize
new migrate_tools-batch-rollback-reroll-for-6_0_x-3014353-11.patch7.29 KB

I have rerolled the patch for version 6.0.x

ygstnd’s picture

ygstnd commented
StatusFileSize
new migrate_tools-batch-rollback-reroll-for-6_0_x-3014353-12.patch7.29 KB

I updated the patch added in comment #11
Deleted the not needed "sprintf('%s::%s', static::class, 'batchProcessImport')" which was breaking Migration Batch execution

heddn’s picture

heddn
Primary language English
Location Illinois
commented
Status: Needs review » Needs work

This could use a re-base and review if everything is in the MR.