Problem/Motivation
The existing Drush commands in the entity_mesh module use drush_batch for processing items. On large sites with many entities to process, this can lead to memory leaks, causing the command to fail and not all items to be processed.
Because the process is handled in a batch, it's not possible to resume the operation from where it left off. If the process fails, the entire task must be restarted from the beginning.
Steps to reproduce
This issue is a feature request, so there are no specific steps to reproduce a bug. The problem is related to the performance of existing Drush commands on large-scale sites.
Proposed resolution
The entity_mesh module already contains a Queue Worker at src/Plugin/QueueWorker/EntityMeshQueueWorker.php which is a more robust solution for processing large numbers of items asynchronously.
The proposed solution is to:
- Create a new Drush command that enqueues all the items to be processed into the queue.
- Users can then process the queue by running drush queue:run entity_mesh or by configuring a cron job to handle the processing automatically.
This approach will:
- Prevent memory issues on large datasets.
- Allow the process to be run in the background.
- Enable the processing to be easily restarted from the last item if an error occurs.
- Leverage the existing queue system within Drupal, which is designed for this type of task.
Remaining tasks
Create the new Drush command that enqueues the items. This will involve writing code to query the relevant entities and add them to the queue for processing by the existing `EntityMeshQueueWorker`.
| Comment | File | Size | Author |
|---|---|---|---|
| #6 | queue_process.png | 177.85 KB | eduardo morales alberti |
Issue fork entity_mesh-3544383
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #3
lpeidro commentedComment #5
eduardo morales albertiComment #6
eduardo morales albertiThere is a problem with the memory, we tried to reset the node cache, but it still increments with each item, so we should review how to solve it.
Comment #7
lpeidro commentedI'm closing this issue as a new Drush command has been added to track all the entities to be processed by Entity Mesh: https://www.drupal.org/project/entity_mesh/issues/3544912