Implement running migration processes (especially import and rollback) from the front page and the group page in the UI.

The operations area from the D7 UI for reference:

Operations detail

CommentFileSizeAuthor
#40 2470882-40.patch32.36 KBheddn
#40 interdiff_38-40.txt692 bytesheddn
#38 interdiff_30-38.txt24.37 KBheddn
#38 2470882-38.patch32.3 KBheddn
#30 implement_running-2470882-30.patch25.95 KBedurenye
#30 interdiff-implement_running-2470882-29-30.txt1.46 KBedurenye
#29 implement_running-2470882-29.patch25.64 KBedurenye
#23 interdiff-2470882-22-23.txt657 bytesZemelia
#23 migrate_tools-migrate_ui-2470882-23.patch26.18 KBZemelia
#22 migrate_tools-migrate_ui-2470882-22.patch26.15 KBZemelia
#22 interdiff-2470882-21-22.txt754 bytesZemelia
#21 migrate_tools-migrate_ui-2470882-21.patch26.2 KBZemelia
#20 migrate_tools-migrate_ui-2470882-20.patch29.22 KBZemelia
#19 interdiff-2470882-18-19.txt32.23 KBZemelia
#19 migrate_tools-migrate_ui-2470882-19.patch39.41 KBZemelia
#18 migrate_tools-migrate_ui-2470882-18.patch49.53 KBkriboogh
#18 interdiff-2470882-16-18.txt9.38 KBkriboogh
#16 migrate_tools-running-migration-processes-ui-2470882-16.patch42.86 KBkriboogh
#16 interdiff-2470882-13-16.txt44.76 KBkriboogh
#13 migrate_tools-running-migration-processes-ui-2470882-13.patch15.15 KBkriboogh
#12 migrate_tools-running-migration-processes-ui-2470882-12.patch15.43 KBkriboogh
#11 migrate_tools-ui-runner-2470882-11.patch2.98 KB8ballsteve
#9 interdiff-2470882-7-9.txt882 bytespguillard
#9 running_migration_processes_ui-2470882-9.patch10.87 KBpguillard
#7 interdiff-2470882-2-7.patch8.47 KBpguillard
#7 running_migration_processes_ui-2470882-7.patch10.85 KBpguillard
#2 running_migration_processes_ui-2470882-2.txt11.11 KBpguillard
#2 launch.png88.02 KBpguillard
#2 migrations.png61.48 KBpguillard
Operations_detail.jpg115.88 KBmikeryan
Members fund testing for the Drupal project. Drupal Association Learn more

Comments

mikeryan’s picture

Project: Migrate Plus » Migrate Tools
pguillard’s picture

Version: » 8.x-1.x-dev
Status: Active » Needs review
FileSize
61.48 KB
88.02 KB
11.11 KB

Here is a patch :

It contains a new route admin/config/migrate/migrations/launch/{migration}

I copied and adapted to D8 the form (From migrate_ui.pages.inc from Migrates D7). It looks like this :
Launch

I did not include the top selection part, but I tried to adapt to the new UI with new launch buttons :
Migrations

The submitForm() function needs to be updated, some D7 code is there. I could not convert it to D8, need to know more about D8 Batch API...

Any Thoughts?...

draenen’s picture

Status: Needs review » Needs work

Is your patch based off (or intended for) the migrate_ui module? The route configuration should be in migrate_tools.routing.yml instead of migrate_ui.routing.yml.

pguillard’s picture

Yes I guess this is migrate_ui stuff. I guess I should move it to migrate_ui project !?

mikeryan’s picture

Yes I guess this is migrate_ui stuff. I guess I should move it to migrate_ui project !?

My understanding is that @benjy created migrate_ui to hold the editing UI for migrations (he feels that the editing UI should be separate from the UI for running migrations). I, personally, would love to see your runner patch done against the migrate_tools UI.

arknoll’s picture

Any more work being done on this patch?

pguillard’s picture

Status: Needs work » Needs review
FileSize
10.85 KB
8.47 KB

I had stuck in my old patch, sorry, and it took time to get back...

Here is a patch with the form intended for migrate_tools, this time !

The form will not submit yet, I had troubles to adapt these parts from Migrate D7:

  • Error: Call to undefined function drush_get_option()
  • The migrate_drush_path variable
mikeryan’s picture

Let's not worry about the drush support at this point, that should be a separate followup issue - just take out anything referencing background operations and drush.

pguillard’s picture

So I just commented everything concerning drush in the submitForm.

mikeryan’s picture

Status: Needs review » Needs work

Finally took a good look at this - unfortunately, I think it needs to go back to the drawing board. For one thing, this really should be set up for bulk operations - the ability to select multiple migrations and run them at once. Doing it one at a time, and having to go through a secondary form to select the operation and execute the migrations, is simply going to be too tedious. Another thing is that it should not be calling drush functions - it should have its own batch code for executing the migrations.

8ballsteve’s picture

I made an initial pass at exposing a new route and "run" method on the "process" tab to process these through the UI using the batch API. It works but doesn't utilise the Batch API very elegantly - simply bundles the whole process into a single operation but might be a useful starting point for someone else?

Cheers
Steve

kriboogh’s picture

Here is a patch against '8.x-3.0-beta1'. It uses the work from both #11 and #9 but doesn't rely on drush code. Still needs work though . On of the issues this has, is apparently the event listeners on postImport are not called when the import is done, making the last migration date not being set.

kriboogh’s picture

Found the reason the listeners weren't called.

mstrelan’s picture

I tested the patch in #13 and found the limit doesn't seem to do anything, and the batch seems to import everything at once rather than in batches. Other than that, the Import and Rollback operations appeared to work as expected. Is there anything else still outstanding for this?

kriboogh’s picture

Having a really quick glance at the code here, what should probably still need to be done is, implement a MigrationBatchExecutable which extends from the MigrateExecutable class. In there we will need to put some code in place that updates the batch process, probably by copy pasting the code from the MigrateExecutableBase functions (for example the import and rollback functions) and splitting/modifiying those so that the individual row imports can update the batch process.

kriboogh’s picture

Here is a first run at this. Due to limitations in how migrate core is implemented (executables can not be serialize, because the source or destination might indirectly hold a reference to a database connection), I had to figure out how the original executable can be called, without having to rewrite (copy-paste) the base class code. So what it does now is, a MigrateBatchExecutable extends from the migrate_tools/MigrateExecutable and sets up a (bunch of) batch operations for the current migration. The batch process is then fired and during each run, the executable is re-created and re-initialised. During each run, the core migration deals with the housekeeping of the migration (skipping rows already imported in the previous run), the checkStatus method checks if the amount of items to be processed is reached and terminates the current run. Batch API will see it is not done yet and will initiate the next run. All this time batch context is used to keep track of were we are in the import process.
It still needs some tweaking, such as more useful error tracking in case something goes wrong. Also the batch process is currently sliced into 1% runs. So each batch run will handle (total_items_to_import / 100) items. For small batches this might be to restrict as it will handle 1 item each run, which is a overhead if you can do them all i one run (but then the progressbar won't update until they are all done, so...)

kriboogh’s picture

Version: 8.x-1.x-dev » 8.x-3.x-dev

changed to the correct dev version.

kriboogh’s picture

Small update that fixes the error notification when broken migrations are present (for example database not available). This caused the overview page not to render.
Also fixed a small update issue on the progress bar of the batch.

Zemelia’s picture

Version: 8.x-3.x-dev » 8.x-4.x-dev
Status: Needs work » Needs review
FileSize
39.41 KB
32.23 KB

Patch for 8.x-4.0-beta1 with https://www.drupal.org/node/2825846 changes.

Zemelia’s picture

FileSize
29.22 KB

Previous patch is wrong as it was based on 8.x-4.0-beta1, patch for 8.x-4.x-dev applied

Zemelia’s picture

FileSize
26.2 KB

Previous patch had duplicates of "migrate_tools.launch" route from previous patch(https://www.drupal.org/files/issues/migrate_tools-migrate_ui-2470882-18....) reroll. Fixed

Zemelia’s picture

FileSize
754 bytes
26.15 KB

Also have spotted issue with infinity batch and limit value.
Patch updated with interdiff.

Zemelia’s picture

FileSize
26.18 KB
657 bytes

Another issue noticed:
Notice: Undefined index: sandbox in Drupal\migrate_tools\MigrateBatchExecutable->checkStatus() (line 266 of /var/www/html/modules/migrate_tools/src/MigrateBatchExecutable.php)

Patch with interdiff attached.

caldenjacobs’s picture

How should we handle additional parameters (such as '--update', '--limit', etc.) in the UI?

edurenye’s picture

Status: Needs review » Needs work

I agree with #2470882-10: Implement running migration processes through the UI this should be done using bulk operations and actions. And '--update', '--limit', etc should be handled in the action form.

kriboogh’s picture

Thanks for finding the infinite issue.

update and limit are supported in the current patch (#23).
(This does not however solve the bulk migrate issue as mentioned in #10, but is does give us at least some ui possibilities to run migrations).

RedEight’s picture

Some ability to run migrations via ui is better than no ability via ui. Is there any reason we can't get this committed now and open a new issue to add batch support and another to add background processing via drush (similar to how it used to be done https://www.drupal.org/node/1958170)

RedEight’s picture

With the patch in #23 I got the following error

InvalidArgumentException: Malformed UTF-8 characters, possibly incorrectly encoded in Symfony\Component\HttpFoundation\JsonResponse->setData()

I also received the following related error

Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xB5m fil...' for column 'message' at row 1: INSERT INTO {migrate_message_20170807__processhq_items_custom_upload_csv} (source_ids_hash, level, message) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2); Array ( [:db_insert_placeholder_0] => 38aa38535b9c7af8947577caa928006acf5ae44b47e236a70c06cf965d7a3d12 [:db_insert_placeholder_1] => 1 [:db_insert_placeholder_2] => SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xB5m fil...' for column 'field_line_5_value' at row 1: INSERT INTO {node__field_line_5} (entity_id, revision_id, bundle, delta, langcode, field_line_5_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5); Array ( [:db_insert_placeholder_0] => 16784 [:db_insert_placeholder_1] => 16806 [:db_insert_placeholder_2] => phq_item [:db_insert_placeholder_3] => 0 [:db_insert_placeholder_4] => en [:db_insert_placeholder_5] => For IS-2002 0.2�m filter line only ) (/home/joshua/public/processhq.com/public/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php:777) ) in Drupal\migrate\Plugin\migrate\id_map\Sql->saveMessage() (line 658 of /home/joshua/public/processhq.com/public/core/modules/migrate/src/Plugin/migrate/id_map/Sql.php).

The field I'm importing is a plain text field and the source is "For IS-2002 0.2µm filter line only". It doesn't like the µ character.

I assume this is due to a discrepancy between encodings in the content I'm migrating and my Drupal site. The only reason I mention it is because this error does not occur when running the same migration via drush so something being done via the UI is different.

EDIT: Further testing revealed that this does actually occur in drush migrations. It was due to my client sending me one file as latin1 instead of UTF-8. Always check your clients are sending you the right file :)

edurenye’s picture

Just rerolled.

edurenye’s picture

joelpittet’s picture

I love this, even if it's not ideal look, it provides a bit of an easier XDebug session through the UI since my CLI is in a VM.

heddn’s picture

Issue tags: +Needs tests

I'm going to ask for a BTB on this. This is a super important method to run a migration, but we should test it as well.

g089h515r806’s picture

import of #30 works correctly.

heddn’s picture

There's also https://www.drupal.org/node/2905437 that has some interesting ideas that could be merged into this.

edysmp’s picture

I'm testing this and I'm getting a infinite issue when I go to: /admin/structure/migrate/manage/migration_group_test/migrations/migration_test/process/run

/admin/structure/migrate/manage/{migration_group}/migrations/{migration}/launch that is fine.

RedEight’s picture

I'm seeing the same issue as edysmp. Also, /admin/structure/migrate/manage/{migration_group}/migrations/{migration}/process throws a whole slew of errors... I don't think it's super critical, but it is accessible from /admin/structure/migrate/manage/{migration_group}/migrations/{migration} so it would be nice if it didn't. The errors appear to stem from

Warning: strlen() expects parameter 1 to be string, array given in Drupal\Component\Utility\Unicode::validateUtf8() (line 689 of core/lib/Drupal/Component/Utility/Unicode.php).

as the subsequent errors seems to be render complaining about being handed an array key instead of a renderable element. Could this be because I use multiple chained process plugins on an element?

ressa’s picture

I just tested the patch in #30 with my migration and it works fine, thanks for the great work! There wasn't any "Import all" button, but I could run all migrations from the "main migration" by adding "sub-migrations" like this:

migration_dependencies:
  required:
    - a_sub_migration
    - another_sub_migration

EDIT: It seems like putting a number in Limit to: is ignored, and everything is imported ...

Status: Needs review » Needs work

The last submitted patch, 38: 2470882-38.patch, failed testing. View results

heddn’s picture

jdleonard’s picture

Upon applying #38 and running drush cr, should I be able to see the migrations shown when running drush ms at the path "admin/structure/migrate/manage/default/migrations"? I'm not seeing any migrations listed in the UI, but I am via the Drush command.

heddn’s picture

re #41: do the migrations have an assigned migration_group of default?

heddn’s picture

This now has an automated test and I've point/click tested it manually. Can we get some other testing done and I'll get this merged ASAP?

heddn’s picture

This is super cool. I just took a custom json migration's configuration. Added this patch to simplytest.me and was able to suck in several nodes to a site. The link is good for another 23hrs for the proof: https://du29x.ply.st/node

heddn’s picture

Status: Needs review » Reviewed & tested by the community

I'm sure we can do better than what we have here, but let's do that in follow-ups. We have automated tests. I've tested it on local and on simplytest.me. It just works. There's been a fair amount of review over the past couple weeks. I'm about to commit.

edurenye’s picture

RTBC +1 I tested it manually and works fine.

Lets commit it and as you say then we can create a follow up to discuss how we can improve it.

I still don't like how is working and I would change it to use bulk operations. But this is working just fine and it's better than nothing.

  • heddn committed bf98b07 on 8.x-4.x
    Issue #2470882 by Zemelia, kriboogh, pguillard, heddn, edurenye,...
heddn’s picture

Status: Reviewed & tested by the community » Fixed

I've opened up #2924298: Add limit to Web UI execution & #2924296: Batch (VBO) functionality for Web UI executions to address the outstanding issues we didn't include in this initial release. Thanks for everyone's patience over the past 2 years. And let's get the follow-ups moving along too!

jdleonard’s picture

@heddn: Reinstalling my custom migration module resolved the issue I raised. I'm not sure whether it's possible to do some work to avoid that step, but it's not a big deal. Might be worth documenting though. Thanks for the patch!

Topplestack’s picture

I'm getting: Cannot apply patch on #40.

I should mention I've been using #30 and it has been working well.

Topplestack’s picture

Status: Fixed » Needs work

It looks like the update to migrate_tools release this morning and the patch rolled yesterday aren't compatible.

heddn’s picture

Status: Needs work » Fixed

re #50, that's because it is already applied and committed.

ressa’s picture

I just tried the Beer migration example from Migrate Plus via GUI, and it works just perfectly, great work! Here is a simplytest.me link with the required modules: https://simplytest.me/project/migrate_tools/8.x-4.x?add[]=migrate_plus&a...

Enable these modules:

  • Migrate Example
  • Migrate Plus
  • Migrate Tools
  • Admin Toolbar (not required)

Now you can run the "Beers of the world" migration from /admin/structure/migrate/manage/beer/migrations

Note: Executing the "Beers of the world" migration will migrate all three migrations, but you need to roll back each migration individually. This simplytest.me link is good for another 23 hours: https://duukq.ply.st/admin/structure/migrate/manage/beer/migrations

heddn’s picture

As a final word, we just used the dev branch of this code and imported a bunch of migrate_plus config to migrate data using a json source in a lab/training session at Drupal Camp Lagos y Volcanes. Using the web UI was super important for the training because on simplytest.me, we do not have access to drush. This made setup and training to use migrate super easy and accessible. Thanks to simplytest.me and all those who contributed to this patch. The bar for using migrate in D8 is now lowered.

Oh, and everyone one of the migrations on everyone of the students computers worked first time. Shocking. It just worked (tm).

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

marcvangend’s picture