Problem/Motivation
Currently, the way migrate retry module produce bottlenecks processing queued items because:
- The queue worker runs the migration with a specific item, without excluding the rest of items. This means that for each queue item processed, all items in the migration source are being iterated. This number can be high and make the item processing slower.
- As it runs the migration, it must postpone the item if the migration is running. This makes a single item that can be quickly migrated be pending on the migration to stop.
- It process one item per step. Migrations are more performant running multiple items , building a migration for jsut one item is not efficient.
The last point is controversial as queue items should be atomic. But, we are aware this module is being used to queue multiple items at once using the vbo_queue module. So it makes sense process a group of items.
Proposed resolution
- Ensure migration only process queued items in it's source plugin, and not process all the originally queued items.
- Allow enqueue multiple items and configure the limit of items to be queued.
Issue fork migrate_retry-3612489
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3612489-prevent-migrate-retry
changes, plain diff MR !15
Comments
Comment #2
omarlopesinoComment #6
omarlopesinoSolutions are done in this MR