James Nesbitt's sandbox: ImportQueue

An importing/migration module that separates remote data retrieval from Drupal Entity creation/updating by a queue.

• Remote data retrieval, and the corresponding updating of Drupal content itself, can be managed and scheduled separately, in order to optimize them separately;
• Multiple data retrievals can feed the same Drupal content code, to implement different schedules for delta checks, full data update, check for removals;

Following in the lines of feeds, with some concepts from the migrate module, this module could grow to complement both, by becoming a source of data for either. The point of this module is to separate the migration of Drupal entities by first retrieving source data & metadata, and keeping that in queue items. The items can then be used for CRUD operations on Drupal entities.

The module was conceived during a project where multiple remote data sources were being used for large scale repeated (daily) migrations, where the various backends were of varied nature and reliability. The overall approach was used to maximize stability of the migrations, and give maximum control over batching to schedulers. The definition of the remote data, the retrieval of that data, and the processing of that data can all be defined and controlled separately.

The Importing is performed in 2 steps:

1. Remote retrieval of data, which is used to create Queue items (this process is called item tasking)
2. Processing of Queue items to relate to Drupal entities (calling item processing)

What is has:
- entirely ctools plugin based; plugins are mostly info based, with dependency processing built in. Plugins may implement easy to use configuration and handling;
- Batching for tasks and queue processing;
- Drush implementation for batching;
- Fancy admin UI for plugin configuration, queue management, task batching and item batching;
- Generous and plugin based operation Logging that can easily be adapted for any needs;
- Completely customizable logging interface, configurable in the UI.
- Plenty of functionality to customize batch resume, and batch fail;

What it needs:
- more advanced queue control;

What it should have:
- Feed source integration: This module should be able to push queue items into the feeds module;
- Migrate module source implementation: This modules should be able to push items into the Migrate module;
- better abstraction of data and metadata, which could be used for mapping fields in feeds and migrate module.

Some Notes:
- Initially we used DrupalQueue for queue handling, but it wasn't enough (no listing of items in queue, no retention of items after processing) so now two custom queue handling plugins are offered, one DB based, and one file based. The file based approach has advantages for really large queues, as long as you are not worried about duplicates. The DB queue is the go-to standard.
- The module is not getting a lot of attention right now, as Drupal8 will require re-writing the OOP code. It's fed a lot of ideas that can easily be used in D8, and may get reborn, if there is a need for it. If you need maintenance. please feel free to ask.

Initial Effort by:
https://www.drupal.org/u/jaxxed (author alter-ego)

and lots of support by other Wunderkraut (BE, UK, LV) devs