Implement rollback of items no longer in source data

As long as the other patch gets in, this one seems to be working well:) Thanks @kevin.dutra

Blocked on the core issue.

hello, guys! :)

Since #2914005: Drush 9 port of commands went into Migrate tools, I believe the patch here from comment #2 lacks support for that Drush 9 part. I am attaching a patch that keeps all of kevin.dutra's work + support for Drush 9 with the same functionality/interface (--missing-from-source). I've tested it locally on Drush 9.

I am moving the issue status to "needs review" since there's a new patch to review (I assume if my patch is acceptable, then we can move it back to postponed until the core catches up).

Also, I believe the targeted version is 8.x-4.x, isn't it? At least my patch is against 8.x-4.x.

Needed a re-roll because of some short array syntax changes in migrate_tools.drush.inc

Thanks for your code, if you want to remove missing items from code you can use:

$migration = \Drupal::service('plugin.manager.migration')->createInstance($migration);
      $executable = new MigrateExecutable(
        $migration, 
        new MigrateMessage()
      );
      $executable->rollbackMissingItems();

Example: How to start a migration programmatically

Would love to see this for Drupal 7 too.

@jeff.hartman I think this issue for D7 #1416672: Optionally delete destination when source data is removed (with tests)

Re-rolled

Waiting on #2809433: Migrate support for deleting items no longer in the incoming data to land.

I needed this while we're still waiting for the other issue, so I re-rolled the patch. Up to y'all if you want to set it to "Needs review"; not gonna interfere with the issue status.

another patch reroll

I'm interested in using the "limit" option while rolling back missing items, as processing large numbers of rows on cron will cause it to timeout. I assume migrate_tools may need to override migrateMissingItems() once #2809433 is merged. Is that correct? For now, I'll be adding that function from the core issue to a custom module and overriding it there.

Reroll

Patch rerolled for migrate_tools 4.0 on Drupal 8.6.7

Patch rolled for migrate_tools 4.1

Note that this patch doesn't work well and does nothing if you don't apply also the core patch from #2809433: Migrate support for deleting items no longer in the incoming data. This patch only exposes the new funcionality added in that other patch through the drush command migrate:rollback

The last patch was not applying anymore, I have just re-rolled it.

It is important to notice that even if this patch was written when the related #2809433: Migrate support for deleting items no longer in the incoming data was targeting core, the method in migrate_tools executable have not
changed its name, and therefore the changes here are still valid.

Re-rolled!

My bad, @marvil07's patch works fine.

Can someone make me understand what wrong I have done while creating the patch ?

I can see some ambiguity in code: https://git.drupalcode.org/project/migrate_tools/blob/8.x-4.x/migrate_to... and the above patch.

Line #79 is different in both code and patch, even though the patch is created from 8.x-4.x branch.

Noting that my experience here is that the patch in #22 does not apply to 4.1.0 – only to 4.x-dev.

For what it's worth, the following patches are playing well together after going from Drupal 8.5 to 8.7.3.
It took me some time to find a working combination.

"drupal/core": {
  "Correct migration lookup to work with multiple values": "https://www.drupal.org/files/issues/2018-12-31/drupal-migrate_lookup_multiple_ids-2890844-43.patch",
},
"drupal/migrate_plus": {
  "Handle XML child nodes as XML in xml-based migrations": "https://www.drupal.org/files/issues/migrate_plus_xml_return_asxml.patch"
},
"drupal/migrate_tools": {
  "Introduce drush switch for rolling back missing migration items": "https://www.drupal.org/files/issues/2019-01-22/migrate-tools-rollback-missing-items-from-source-2863426-20.patch",
  "Migrate support for deleting items no longer in the incoming data": "https://www.drupal.org/files/issues/2019-04-22/2809433-136.patch"
}

As commented in the sister story, https://www.drupal.org/project/migrate_tools/issues/2809433#comment-1317..., this works for us with #20 and 8.x-4.1.

Thanks!

I was having problems running the rollback command more than once. It would work after clearing the cache, but subsequent runs would fail with the error You must specify --all, --group, --tag, or one or more migration names separated by commas.
It turns out the the order or the command line arguments is important. The correct usage is migrate-rollback beer_term --missing-from-source.

Here's a patch that updates the usage documentation.

I tried to use this patch with Drush 8 and while the patch looks good and it applies without error, it seems that git still fails to apply it correctly to the latest 8.x-4.x-dev. git merges the new option into the arguments section of the rollback command instead of the options section, resulting in a Unknown option: --missing-from-source. error when you try to use it. Attached is an updated version of the patch that should fix this.

Updated the patch for 8.x-4.5.

This could be closed duplicate or closed won't fix. See #2809433: Migrate support for deleting items no longer in the incoming data. Closing dup.

The patch in #31 didn't work for me with 8.x-4.5. New patch attached.

Hello, the patch in #33 didn't work for me with 8.x-5.0.0 due to missing migrate_tools.drush.inc file.

Updated patch against latest version of the module

I havent had a chance to test the work yet, but this is a port of the work in #36 to work with the latest 5.0 version.

Bare in mind that you'll need this patch also, for this to work:

https://www.drupal.org/project/migrate_tools/issues/2809433#comment-1307...

This migrate_tools_2863426.patch doesn't apply anymore when I upgrade to 6.0 version for D10 compatibility. I updated the patch for version 6.0.

MigrateToolsCommands.php is moved in 6.0.2 from src/Commands/ to src/Drush/

Update patch for version 6.0.4.