Expose the ability to rollback items no longer included in source data to Drush once #2809433: Migrate support for deleting items no longer in the incoming data is complete.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

kevin.dutra created an issue. See original summary.

kevin.dutra’s picture

Based on the current direction, this is what would be needed.

joelpittet’s picture

Status: Needs review » Reviewed & tested by the community

As long as the other patch gets in, this one seems to be working well:) Thanks @kevin.dutra

mikeryan’s picture

Status: Reviewed & tested by the community » Postponed

Blocked on the core issue.

bucefal91’s picture

Version: 8.x-3.x-dev » 8.x-4.x-dev
Status: Postponed » Needs review
FileSize
1.96 KB
4.31 KB

hello, guys! :)

Since #2914005: Drush 9 port of commands went into Migrate tools, I believe the patch here from comment #2 lacks support for that Drush 9 part. I am attaching a patch that keeps all of kevin.dutra's work + support for Drush 9 with the same functionality/interface (--missing-from-source). I've tested it locally on Drush 9.

I am moving the issue status to "needs review" since there's a new patch to review (I assume if my patch is acceptable, then we can move it back to postponed until the core catches up).

Also, I believe the targeted version is 8.x-4.x, isn't it? At least my patch is against 8.x-4.x.

Peacog’s picture

Needed a re-roll because of some short array syntax changes in migrate_tools.drush.inc

aperedos’s picture

Thanks for your code, if you want to remove missing items from code you can use:

$migration = \Drupal::service('plugin.manager.migration')->createInstance($migration);
      $executable = new MigrateExecutable(
        $migration, 
        new MigrateMessage()
      );
      $executable->rollbackMissingItems();

Example: How to start a migration programmatically

jeff.hartman’s picture

Would love to see this for Drupal 7 too.

Majdi’s picture

heddn’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll
Peacog’s picture

Status: Needs work » Needs review
FileSize
4.62 KB

Re-rolled

rpayanm’s picture

Issue tags: -Needs reroll
heddn’s picture

Status: Needs review » Postponed
wizonesolutions’s picture

I needed this while we're still waiting for the other issue, so I re-rolled the patch. Up to y'all if you want to set it to "Needs review"; not gonna interfere with the issue status.

lathan’s picture

byrond’s picture

I'm interested in using the "limit" option while rolling back missing items, as processing large numbers of rows on cron will cause it to timeout. I assume migrate_tools may need to override migrateMissingItems() once #2809433 is merged. Is that correct? For now, I'll be adding that function from the core issue to a custom module and overriding it there.

KarlShea’s picture

achraf.noomane’s picture

Patch rerolled for migrate_tools 4.0 on Drupal 8.6.7

achraf.noomane’s picture

Patch rolled for migrate_tools 4.1

vidorado’s picture

Note that this patch doesn't work well and does nothing if you don't apply also the core patch from #2809433: Migrate support for deleting items no longer in the incoming data. This patch only exposes the new funcionality added in that other patch through the drush command migrate:rollback

marvil07’s picture

The last patch was not applying anymore, I have just re-rolled it.

It is important to notice that even if this patch was written when the related #2809433: Migrate support for deleting items no longer in the incoming data was targeting core, the method in migrate_tools executable have not
changed its name, and therefore the changes here are still valid.

navneet0693’s picture

FileSize
4.43 KB

Re-rolled!

navneet0693’s picture

My bad, @marvil07's patch works fine.

navneet0693’s picture

Can someone make me understand what wrong I have done while creating the patch ?

I can see some ambiguity in code: https://git.drupalcode.org/project/migrate_tools/blob/8.x-4.x/migrate_to... and the above patch.

Line #79 is different in both code and patch, even though the patch is created from 8.x-4.x branch.

jds1’s picture

Noting that my experience here is that the patch in #22 does not apply to 4.1.0 – only to 4.x-dev.

agileadam’s picture

For what it's worth, the following patches are playing well together after going from Drupal 8.5 to 8.7.3.
It took me some time to find a working combination.

"drupal/core": {
  "Correct migration lookup to work with multiple values": "https://www.drupal.org/files/issues/2018-12-31/drupal-migrate_lookup_multiple_ids-2890844-43.patch",
},
"drupal/migrate_plus": {
  "Handle XML child nodes as XML in xml-based migrations": "https://www.drupal.org/files/issues/migrate_plus_xml_return_asxml.patch"
},
"drupal/migrate_tools": {
  "Introduce drush switch for rolling back missing migration items": "https://www.drupal.org/files/issues/2019-01-22/migrate-tools-rollback-missing-items-from-source-2863426-20.patch",
  "Migrate support for deleting items no longer in the incoming data": "https://www.drupal.org/files/issues/2019-04-22/2809433-136.patch"
}
damontgomery’s picture

As commented in the sister story, https://www.drupal.org/project/migrate_tools/issues/2809433#comment-1317..., this works for us with #20 and 8.x-4.1.

Thanks!

Peacog’s picture

I was having problems running the rollback command more than once. It would work after clearing the cache, but subsequent runs would fail with the error You must specify --all, --group, --tag, or one or more migration names separated by commas.
It turns out the the order or the command line arguments is important. The correct usage is migrate-rollback beer_term --missing-from-source.

Here's a patch that updates the usage documentation.

FeyP’s picture

FileSize
854 bytes
25.32 KB

I tried to use this patch with Drush 8 and while the patch looks good and it applies without error, it seems that git still fails to apply it correctly to the latest 8.x-4.x-dev. git merges the new option into the arguments section of the rollback command instead of the options section, resulting in a Unknown option: --missing-from-source. error when you try to use it. Attached is an updated version of the patch that should fix this.

lammensj’s picture

FileSize
25.3 KB

Updated the patch for 8.x-4.5.

heddn’s picture

Status: Postponed » Closed (duplicate)

This could be closed duplicate or closed won't fix. See #2809433: Migrate support for deleting items no longer in the incoming data. Closing dup.

john.oltman’s picture

FileSize
4.44 KB

The patch in #31 didn't work for me with 8.x-4.5. New patch attached.

fabienly’s picture

Hello, the patch in #33 didn't work for me with 8.x-5.0.0 due to missing migrate_tools.drush.inc file.

visantinian’s picture

FileSize
4.48 KB

Updated patch against latest version of the module

visantinian’s picture

FileSize
4.48 KB
ras-ben’s picture

FileSize
2.13 KB

I havent had a chance to test the work yet, but this is a port of the work in #36 to work with the latest 5.0 version.

Bare in mind that you'll need this patch also, for this to work:

https://www.drupal.org/project/migrate_tools/issues/2809433#comment-1307...

pilot3’s picture

This migrate_tools_2863426.patch doesn't apply anymore when I upgrade to 6.0 version for D10 compatibility. I updated the patch for version 6.0.

Alsén’s picture

MigrateToolsCommands.php is moved in 6.0.2 from src/Commands/ to src/Drush/

Alsén’s picture

Version: 8.x-4.x-dev » 6.0.4
FileSize
2.17 KB

Update patch for version 6.0.4.