Problem/Motivation

Currently, the way migrate retry module produce bottlenecks processing queued items because:

  • The queue worker runs the migration with a specific item, without excluding the rest of items. This means that for each queue item processed, all items in the migration source are being iterated. This number can be high and make the item processing slower.
  • As it runs the migration, it must postpone the item if the migration is running. This makes a single item that can be quickly migrated be pending on the migration to stop.
  • It process one item per step. Migrations are more performant running multiple items , building a migration for jsut one item is not efficient.

The last point is controversial as queue items should be atomic. But, we are aware this module is being used to queue multiple items at once using the vbo_queue module. So it makes sense process a group of items.

Proposed resolution

  • Ensure migration only process queued items in it's source plugin, and not process all the originally queued items.
  • Allow enqueue multiple items and configure the limit of items to be queued.
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

omarlopesino created an issue. See original summary.

omarlopesino’s picture

Assigned: Unassigned » omarlopesino

  • omarlopesino committed a587f160 on 2.0.x
    Issue #3612489: Reimport only the queued rows via the new source and...
omarlopesino’s picture

Status: Active » Fixed

Solutions are done in this MR

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.