Problem/Motivation

The drush radioactivity:fix-references command needs to be batched. It can easily run out of memory when there are hundreds or thousands of missing references. Not that uncommon when installing to established sites or after migrations.

Proposed resolution

Use the BatchBuilder to process entities in a customizable batch size.

CommentFileSizeAuthor
#21 Screenshot 2025-08-30 at 3.12.44 AM.png430.55 KBloze
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

robphillips created an issue. See original summary.

bobooon’s picture

Status: Active » Needs review
bobooon’s picture

Assigned: bobooon » Unassigned

TR made their first commit to this issue’s fork.

chriseikelboom’s picture

Currently having the same issue where it needs to update about ~12000 nodes. The process seems to take about 20-50min (with no feedback). I would love to see this as a batch processor so we can run this process in the background on a production server without possibly causing an interruption of the service or slowing the website to a halt.

thomasdegraaff’s picture

Looking at the code it seems that the dependency injection for the entity type manager service is removed, and the existing dependency injection for the logger service and the radioactivity reference updater service is not used anymore.

Dependency injection is the preferred method for accessing and using services in Drupal 8 and should be used whenever possible. Is there a reason not to use dependency injection in this case?

bobooon’s picture

DI is preferred when not using static methods. Batch API requires operation callbacks to be static methods or functions.

tr’s picture

@Tr4nzNRG
@thomasdegraaff

Please apply the patch and test it, then report your results here. The patch looks good to me, but I don't have a site with a lot of radioactivity nodes I can test it on. I would prefer that at least one person tests this patch on real data before I commit it it's committed. (EDIT: I forgot, I'm not a maintainer on this project).

chriseikelboom’s picture

- removed -

chriseikelboom’s picture

I tested this on real data in my DDEV environment it seems to works as intended. For now this could be merged with the dev? @thomasdegraaff

However I think it still needs a minor improvement. I noticed that the batch process seems to slow down after 2000-5000 nodes. The reason why I don't know yet as the website I'm using has some complexity with other modules and content (direct indexing with Search API/Solr, cache invalidation?).

When I stop the process and restart the command it's back at it's original speed. So maybe this could be changed in the future? For now this is already a good improvement and allows us to use this module for production. So thanks for all the effort so far ;)

chriseikelboom’s picture

Short update: On our TEST environment we had an incident that the webserver ran out of memory. However we aren't sure if this is due this fix or that we have to many other running tasks(Search API indexing, Node Revision Delete). Also our TEST server has less memory than our PROD server.

Coming week we try again. Maybe we let first the other tasks run before update the references.

chriseikelboom’s picture

We used this batch process and noticed that it slows down after processing ~1000/2000 nodes. Manually stopping this process and restarting 'fix' the issue and prevents a out-of-memory (depending on availability from webserver).

Another note and maybe good to add to the documentation when processing large amounts of nodes on high traffic website is that this process in combination with other modules could cause a cascade. For example when used with Node Revision Delete or executing this command while a re-indexing of Search API could cause a out-of-memory depending on the availability of memory from the webserver.

So it's wise to execute this batch/command when other processes aren't running and monitor the process. Like after a build process (release).

In short...

It might be good to adjust this solution so it 'restarts' after ~1000 nodes or find a solution for the OOM issue. For us it was the ?only? methode right now to handle a large amount of nodes on a medium/high traffic website.

tr’s picture

Component: Code » Drush
tr’s picture

Version: 4.0.x-dev » 4.1.x-dev
Status: Needs review » Needs work

I will work on this after #3503754: Update Drush commands for Drush 13+ is committed. The MR will need to be rebased to 4.1.x.

loze made their first commit to this issue’s fork.

loze changed the visibility of the branch 3348339-batch-fix-references to hidden.

loze changed the visibility of the branch 4.0.x to hidden.

loze changed the visibility of the branch 3348339-batch-fix-references to active.

loze’s picture

Status: Needs work » Needs review
StatusFileSize
new430.55 KB

I tried to get this to work with 4.1.x.

It does appear to be batching the command now.

MR46

loze’s picture

MR46 is the latest against 4.1.x I was unable to update the version for MR9

this should be ready to go.