Implement running migration processes (especially import and rollback) from the front page and the group page in the UI.

The operations area from the D7 UI for reference:

Operations detail

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

mikeryan’s picture

Project: Migrate Plus » Migrate Tools
pguillard’s picture

Version: » 8.x-1.x-dev
Status: Active » Needs review
FileSize
61.48 KB
88.02 KB
11.11 KB

Here is a patch :

It contains a new route admin/config/migrate/migrations/launch/{migration}

I copied and adapted to D8 the form (From migrate_ui.pages.inc from Migrates D7). It looks like this :
Launch

I did not include the top selection part, but I tried to adapt to the new UI with new launch buttons :
Migrations

The submitForm() function needs to be updated, some D7 code is there. I could not convert it to D8, need to know more about D8 Batch API...

Any Thoughts?...

draenen’s picture

Status: Needs review » Needs work

Is your patch based off (or intended for) the migrate_ui module? The route configuration should be in migrate_tools.routing.yml instead of migrate_ui.routing.yml.

pguillard’s picture

Yes I guess this is migrate_ui stuff. I guess I should move it to migrate_ui project !?

mikeryan’s picture

Yes I guess this is migrate_ui stuff. I guess I should move it to migrate_ui project !?

My understanding is that @benjy created migrate_ui to hold the editing UI for migrations (he feels that the editing UI should be separate from the UI for running migrations). I, personally, would love to see your runner patch done against the migrate_tools UI.

arknoll’s picture

Any more work being done on this patch?

pguillard’s picture

Status: Needs work » Needs review
FileSize
10.85 KB
8.47 KB

I had stuck in my old patch, sorry, and it took time to get back...

Here is a patch with the form intended for migrate_tools, this time !

The form will not submit yet, I had troubles to adapt these parts from Migrate D7:

  • Error: Call to undefined function drush_get_option()
  • The migrate_drush_path variable
mikeryan’s picture

Let's not worry about the drush support at this point, that should be a separate followup issue - just take out anything referencing background operations and drush.

pguillard’s picture

So I just commented everything concerning drush in the submitForm.

mikeryan’s picture

Status: Needs review » Needs work

Finally took a good look at this - unfortunately, I think it needs to go back to the drawing board. For one thing, this really should be set up for bulk operations - the ability to select multiple migrations and run them at once. Doing it one at a time, and having to go through a secondary form to select the operation and execute the migrations, is simply going to be too tedious. Another thing is that it should not be calling drush functions - it should have its own batch code for executing the migrations.

8ballsteve’s picture

I made an initial pass at exposing a new route and "run" method on the "process" tab to process these through the UI using the batch API. It works but doesn't utilise the Batch API very elegantly - simply bundles the whole process into a single operation but might be a useful starting point for someone else?

Cheers
Steve

kriboogh’s picture

Here is a patch against '8.x-3.0-beta1'. It uses the work from both #11 and #9 but doesn't rely on drush code. Still needs work though . On of the issues this has, is apparently the event listeners on postImport are not called when the import is done, making the last migration date not being set.

kriboogh’s picture

Found the reason the listeners weren't called.

mstrelan’s picture

I tested the patch in #13 and found the limit doesn't seem to do anything, and the batch seems to import everything at once rather than in batches. Other than that, the Import and Rollback operations appeared to work as expected. Is there anything else still outstanding for this?

kriboogh’s picture

Having a really quick glance at the code here, what should probably still need to be done is, implement a MigrationBatchExecutable which extends from the MigrateExecutable class. In there we will need to put some code in place that updates the batch process, probably by copy pasting the code from the MigrateExecutableBase functions (for example the import and rollback functions) and splitting/modifiying those so that the individual row imports can update the batch process.

kriboogh’s picture

Here is a first run at this. Due to limitations in how migrate core is implemented (executables can not be serialize, because the source or destination might indirectly hold a reference to a database connection), I had to figure out how the original executable can be called, without having to rewrite (copy-paste) the base class code. So what it does now is, a MigrateBatchExecutable extends from the migrate_tools/MigrateExecutable and sets up a (bunch of) batch operations for the current migration. The batch process is then fired and during each run, the executable is re-created and re-initialised. During each run, the core migration deals with the housekeeping of the migration (skipping rows already imported in the previous run), the checkStatus method checks if the amount of items to be processed is reached and terminates the current run. Batch API will see it is not done yet and will initiate the next run. All this time batch context is used to keep track of were we are in the import process.
It still needs some tweaking, such as more useful error tracking in case something goes wrong. Also the batch process is currently sliced into 1% runs. So each batch run will handle (total_items_to_import / 100) items. For small batches this might be to restrict as it will handle 1 item each run, which is a overhead if you can do them all i one run (but then the progressbar won't update until they are all done, so...)

kriboogh’s picture

Version: 8.x-1.x-dev » 8.x-3.x-dev

changed to the correct dev version.

kriboogh’s picture

Small update that fixes the error notification when broken migrations are present (for example database not available). This caused the overview page not to render.
Also fixed a small update issue on the progress bar of the batch.

Zemelia’s picture

Version: 8.x-3.x-dev » 8.x-4.x-dev
Status: Needs work » Needs review
FileSize
39.41 KB
32.23 KB

Patch for 8.x-4.0-beta1 with https://www.drupal.org/node/2825846 changes.

Zemelia’s picture

FileSize
29.22 KB

Previous patch is wrong as it was based on 8.x-4.0-beta1, patch for 8.x-4.x-dev applied

Zemelia’s picture

FileSize
26.2 KB

Previous patch had duplicates of "migrate_tools.launch" route from previous patch(https://www.drupal.org/files/issues/migrate_tools-migrate_ui-2470882-18....) reroll. Fixed

Zemelia’s picture

FileSize
754 bytes
26.15 KB

Also have spotted issue with infinity batch and limit value.
Patch updated with interdiff.

Zemelia’s picture

FileSize
26.18 KB
657 bytes

Another issue noticed:
Notice: Undefined index: sandbox in Drupal\migrate_tools\MigrateBatchExecutable->checkStatus() (line 266 of /var/www/html/modules/migrate_tools/src/MigrateBatchExecutable.php)

Patch with interdiff attached.

caldenjacobs’s picture

How should we handle additional parameters (such as '--update', '--limit', etc.) in the UI?

edurenye’s picture

Status: Needs review » Needs work

I agree with #2470882-10: Implement running migration processes through the UI this should be done using bulk operations and actions. And '--update', '--limit', etc should be handled in the action form.

kriboogh’s picture

Thanks for finding the infinite issue.

update and limit are supported in the current patch (#23).
(This does not however solve the bulk migrate issue as mentioned in #10, but is does give us at least some ui possibilities to run migrations).

RedEight’s picture

Some ability to run migrations via ui is better than no ability via ui. Is there any reason we can't get this committed now and open a new issue to add batch support and another to add background processing via drush (similar to how it used to be done https://www.drupal.org/node/1958170)

RedEight’s picture

With the patch in #23 I got the following error

InvalidArgumentException: Malformed UTF-8 characters, possibly incorrectly encoded in Symfony\Component\HttpFoundation\JsonResponse->setData()

I also received the following related error

Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xB5m fil...' for column 'message' at row 1: INSERT INTO {migrate_message_20170807__processhq_items_custom_upload_csv} (source_ids_hash, level, message) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2); Array ( [:db_insert_placeholder_0] => 38aa38535b9c7af8947577caa928006acf5ae44b47e236a70c06cf965d7a3d12 [:db_insert_placeholder_1] => 1 [:db_insert_placeholder_2] => SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xB5m fil...' for column 'field_line_5_value' at row 1: INSERT INTO {node__field_line_5} (entity_id, revision_id, bundle, delta, langcode, field_line_5_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5); Array ( [:db_insert_placeholder_0] => 16784 [:db_insert_placeholder_1] => 16806 [:db_insert_placeholder_2] => phq_item [:db_insert_placeholder_3] => 0 [:db_insert_placeholder_4] => en [:db_insert_placeholder_5] => For IS-2002 0.2�m filter line only ) (/home/joshua/public/processhq.com/public/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php:777) ) in Drupal\migrate\Plugin\migrate\id_map\Sql->saveMessage() (line 658 of /home/joshua/public/processhq.com/public/core/modules/migrate/src/Plugin/migrate/id_map/Sql.php).

The field I'm importing is a plain text field and the source is "For IS-2002 0.2µm filter line only". It doesn't like the µ character.

I assume this is due to a discrepancy between encodings in the content I'm migrating and my Drupal site. The only reason I mention it is because this error does not occur when running the same migration via drush so something being done via the UI is different.

EDIT: Further testing revealed that this does actually occur in drush migrations. It was due to my client sending me one file as latin1 instead of UTF-8. Always check your clients are sending you the right file :)