Make migrate:import --migrate-debug compatible with Drush9+

At the moment these hooks are used: hook_drush_help_alter & hook_drush_command_alter.

The proper way to do this with Drush 9 is probably to extend from MigrateToolsCommands. See my and Moshe's comments on the blog:

See https://github.com/drush-ops/drush/pull/3447 for altering the annotations and https://github.com/consolidation/annotated-command/#replace-command-hook for altering the command method itself.

Thanks for the request.

Willing to accept patches for this. I would work on it but I haven't done much Drupal (or migrate) work.

I'm happy to write a patch for this, if it will help others / become co-maintainer as we're doing a lot of migrates now and would be good to have something that we use which can help others. We do use xdebug normally but its a nice thing to have with drush.

Sure if you want to provide a patch and want to help maintain (And hopefully make better with other features) I'd be glad to add you as a co-maintainer.

I started porting this to Drush 9. This patch:

• Adds the drush.services.yml and composer boilerplate.
• Adds a new DrushCommands class.
• Registers the new options, replacing hook_drush_help_alter().
• Makes the kint() dumps work for the migrate:import commands. That was my primary need.

The tricky part was checking the Drush options in the EventSubscriber. Drush 9 doesn't have a global where the command options are available. I chatted with Greg Anderson (a Drush maintainer) on Slack and we came up with the idea of temporarily storing the command options with State API, so they could be read by the EventSubscriber.

Still TO DO:

• This probably warrants an 8.x-2.x branch, rather than try to support Drush 8 and 9 together.
• Make the migration rebuilds work.
• Maybe combine the EventSubscriber and Drush commands in a single class? This was another idea from the Slack chat.
  

  class MigrateDevelCommands extends DrushCommands implements EventSubscriberInterface {
  

  This would remove the need to use State API, but would kinda violate the single-purpose SOLID principle, and it wouldn't be clear which namespace the class should live in.

If you want to use a global rather than replicating all of the options, there actually is a way to do that. The parsed options are available in the config object. You can see what's in the config object via:

drush ev 'return \Drush\Drush::config()->export();'

You can look at one option via:

\Drush\Drush::config()->get('options.migrate-debug');

Aha, thanks. I see the runtime options and args arrays in the output of \Drush\Drush::config()->export();

If you want to log something to the console, there's \Drush\Drush::logger().

This patch is working as expected and allows me to use `--migrate-debug` etc with Drush 9. Awaiting 1 more for RTBC.

No, it still needs work.

The only functionality that has been ported, so far, is the visible debug output.

The migration config reset feature hasn't been ported yet. The patch contains some @todo tags.

@andrewmacpherson, my apologies I'll review the code more thoroughly and see if I can help.

@andrewmacpherson is it possible to release a dev version with this patch in a 8.x-2.x branch? I'm happy to take on one or two features mentioned throughout this post, but am afraid of on big patch trying to do all.

@botris, Of course, but the project maintainer needs to do that..

@botris thanks for updating the task list.

Thanks for the work guys, will get this released in a branch either tomorrow or next week. Sorry, I mean't to be responding to #3021648

This command used to run on Drush 8

drush -l mysite.com mim --migrate-debug --limit=1 --update custom_product_image_step1

I have applied patch #6 with Drush 9 and I get this error:

In MigrateToolsCommands.php line 293:
                                                                                             
  You must specify --all, --group, --tag or one or more migration names separated by commas 

Hey @jcmartinez! I've been caught out by that before - what you're seeing is likely the parameter order, and not specific to migrate_devel or this patch. I think Drush 9 is more specific about order of params.

Try drush -l example.com mim custom_product_image_step1 --migrate-debug --limit=1 --update instead?

Seems like the migrate:import parameters need to come after the migration name.

vagrant@example:/vagrant$ drush help migrate:import
Perform one or more migration processes.

Examples:
  migrate-import --all                              Perform all...                                  
  migrate-import --tag=user,main_content            Import all...
  migrate-import classification,article             Import new terms...
  migrate-import beer_user --limit=2                Import no more...
  migrate-import beer_user --idlist=5               Import the user...
  migrate-import beer_user --limit=50 --feedback=20 Import 50 users...

Arguments:
  [migration_ids] Comma-separated list of migration IDs. 

Options:
  --all[=ALL]                                   Process all migrations.                                      
  --tag[=TAG]                                   A comma-separated...
  --limit[=LIMIT]                               Limit on the number...
  --feedback[=FEEDBACK]                         Frequency of...
  --idlist[=IDLIST]                             Comma-separated list...
  --update[=UPDATE]                             In addition to...
  --force[=FORCE]                               Force an operation...
  --execute-dependencies[=EXECUTE-DEPENDENCIES] Execute all...
  --timestamp[=TIMESTAMP]                       Show progress ending...
  --total[=TOTAL]                               Show total processed...
  --migrate-debug                               Enable Debug Mode
  --migrate-debug-pre                           Enable Debug Mode...

Aliases: mim, migrate-import

Hey @xurizaemon

You are totally right. Changing the order works!

Hello

I have applied the patch: migrate_devel-drush9-2938677-6.patch but still get no --migrate-debug or --migrate-debug-pre when I execute
drush migrate:import --help

Some details of my installation:
Drupal version : 8.8.1
Site URI : http://default
DB driver : mysql
DB hostname : localhost
DB port :
DB username : drupal
DB name : drupal
Database : Connected
Drupal bootstrap : Successful
Default theme : ji_theme
Admin theme : seven
PHP binary : /usr/bin/php7.2
PHP config : /etc/php/7.2/cli/php.ini
PHP OS : Linux
Drush script : /usr/local/bin/drush
Drush version : 9.7.1
Drush temp : /tmp
Drush configs : /home/vagrant/.drush/drush.yml
/var/www/drupalvm/drupal/vendor/drush/drush/drush.yml
/var/www/drupalvm/drupal/drush/drush.yml
Install profile : standard
Drupal root : /var/www/drupalvm/drupal/web
Site path : sites/default
Files, Public : sites/default/files
Files, Temp : sites/default/files/tempy/

Can anyone help?

Thanks

Marijn

Hello for the record, I try to apply the patch but it dont applied. I am using drush 10 :)

edit : But the debug plugin works and it's still some help :)

Error log :

- Installing drupal/migrate_devel (1.2.0-beta1): Loading from cache
- Applying patches for drupal/migrate_devel
    https://www.drupal.org/files/issues/2018-10-08/migrate_devel-drush9-2938677-6.patch (drush 9 support)
   Could not apply patch! Skipping. The error was: Cannot apply patch https://www.drupal.org/files/issues/2018-10-08/migrate_devel-drush9-2938677-6.patch

This is my rush status :

$drush status
 Drupal version   : 8.8.2
 Site URI         : http://default
 DB driver        : pgsql
 DB hostname      : localhost
 DB port          : 5432
 DB username      : bnppre
 DB name          : bnppre_db
 Database         : Connected
 Drupal bootstrap : Successful
 Default theme    : bartik
 Admin theme      : seven
 PHP binary       : /usr/bin/php
 PHP config       : /etc/php.ini
 PHP OS           : Linux
 Drush script     : /home/real-estate/vendor/drush/drush/drush
 Drush version    : 10.2.2
 Drush temp       : /tmp
 Drush configs    : /home/real-estate/vendor/drush/drush/drush.yml
 Install profile  : standard
 Drupal root      : /home/real-estate/web
 Site path        : sites/default
 Files, Public    : sites/default/files
 Files, Temp      : /tmp

@fabienly I got pretty much the same setup and had no problems in applying the patch. I use a lando based setup btw.

Drupal version   : 8.8.2
 Site URI         : http://...
 DB driver        : mysql
 DB hostname      : database
 DB port          : 3306
 DB username      : drupal8
 DB name          : drupal8
 Database         : Connected
 Drupal bootstrap : Successful
 Default theme    : usfq
 Admin theme      : claro
 PHP binary       : /usr/local/bin/php
 PHP config       :
 PHP OS           : Linux
 Drush script     : /app/vendor/drush/drush/drush
 Drush version    : 10.2.2
 Drush temp       : /tmp
 Drush configs    : /app/vendor/drush/drush/drush.yml
                    /app/drush/drush.yml
                    /app/local/drush.yml
 Install profile  : standard
 Drupal root      : /app/web
 Site path        : sites/default
 Files, Public    : sites/default/files
 Files, Private   : /app/web/../_private
 Files, Temp      : /tmp

I have been using this patch for a while, marking it as RTBC.

I've been added as a new maintainer for the module, and I've been working on this today.

I didn't commit my old patch from #6 verbatim. Instead I realized there were ways to simplify it.

Re. #7 & #8 - I figured out that I could detect this module's options in an event subscriber by using \Drush\Drush::config()->get('runtime.options'). That's great; it's no longer using State API to keep track of the options passed to Drush, which felt clunky and fragile.

The commits here make the --migrate-debug and --migrate-debug-pre work for the drush migrate:import command, like patch #6 did. In the interest of getting an alpha release of the 8.x-2.x branch out ASAP, I'm narrowing the scope of the issue to fix those only.

I've manually tested the extra options for drush migrate:import using:

• Drupal 8.9.0 + Drush 9.0.0 + Migrate Tools 5.0.0
• Drupal 8.9.0 + Drush 10.0.0 + Migrate Tools 5.0.0
• Drupal 8.9.0 + Drush 9.0.0 + Migrate Run 8.x-1.0-beta3
• Drupal 8.9.0 + Drush 10.0.0 + Migrate Run 8.x-1.0-beta3

Some issue credits. These people moved the issue forward in a significant way:

• @mpp for linking to the docs about updating to Drush 9.
• @greg.1.anderson (Drush maintainer) for advice about checking Drush options, here and in Slack
• @xurizaemon for helping others use the patch
• @botris for tidying up the issue summary todo list

Updating title to reflect the scope.

Hello,
I can't apply patch :

Gathering patches for dependencies. This might take a minute.
  - Installing drupal/migrate_devel (2.0.0-alpha2): Loading from cache
  - Applying patches for drupal/migrate_devel
    https://www.drupal.org/files/issues/2018-10-08/migrate_devel-drush9-2938677-6.patch (Updates Drush for Migrate Devel commands)
   Could not apply patch! Skipping. The error was: Cannot apply patch https://www.drupal.org/files/issues/2018-10-08/migrate_devel-drush9-2938677-6.patch

Any idea ?

Frédéric

@freed_paris:

If you are using the 2.0.x branch of migrate_devel, you don't need the patch, because that branch is compatible with Drush 9+.

The output you posted comes from running composer install, and trying to apply the patch using cweagans/composer-patches. It shows you are installing migrate_devel 2.0.0-alpha2, and trying to apply patch #6 from this issue.

Try these steps:

1. Edit the composer.json file, to remove the patch for migrate devel.
2. Then run composer require drupal/migrate_devel:^2.0

Hi,

Issue still need to be reviewed. I just installed a fresh Drupal 9.2.6 install along with migrate_devel 8.x-2.0-alpha2, devel 4.1 and issue persists with latest Drush 10.6.0. Tried applying the patch but the error is the same with or without the patch.

 Drupal version   : 9.2.6
 Site URI         : http://default
 DB driver        : mysql
 DB hostname      : localhost
 DB port          : 3306
 DB username      : drupal
 DB name          : mim_drupal9
 Database         : Connected
 Drupal bootstrap : Successful
 Default theme    : bartik
 Admin theme      : seven
 PHP binary       : /usr/bin/php7.4
 PHP config       : /etc/php/7.4/cli/php.ini
 PHP OS           : Linux
 Drush script     : /home/andres/www/mim-drupal9/web/vendor/drush/drush/drush
 Drush version    : 10.6.0
 Drush temp       : /tmp
 Drush configs    : /home/andres/.drush/drush.yml
                    /home/andres/www/mim-drupal9/web/vendor/drush/drush/drush.yml
 Install profile  : standard
 Drupal root      : /home/andres/www/mim-drupal9/web
 Site path        : sites/default
 Files, Public    : sites/default/files
 Files, Temp      : /tmp

This issue still needs to be reviewed. We are attempting to migrate a large site from Drupal 7 - Drupal 9, and are on a fresh Drupal 9.4.7 install along with migrate_devel 8.x-2.0-alpha2, and the same issue persists with Drush 11.4.0. Has anyone found a solution? This bug makes the module essentially useless.

Hi @BThiemeNatlaw

I think it's that the release installed on that site was generated a few days before the commit in this issue was landed.

- 8.x-2.0-alpha2 is dated 23 June 2020.
- Commit fixing this issue was 27 June 2020.

Try composer require 'drupal/migrate_devel:2.x-dev@dev' (or something like that), and see if the issue is resolved please?

There are many benefits to publishing a stable release for a project. There may be things we can do to support an updated release of Migrate Devel, which might in turn reduce the number of people bumping on that issue you appear to have hit.

Practically, I think we largely use Migrate Devel on the 'dev branch' (eg some 2.x-dev build, likely pinning to a specific point to avoid surprises). Since it's a development tooling module, sites may opt to be more tolerant of a dev release.

(There is an issue template for requesting a stable release of a module over in #3239062: 'Stable Release Request' Issue Template which you might find helpful too.)

Hope this sorts it out for you!