At runtime track URLs that have already been queued and skip those.

I think I have a solution for this, a starting point for discussion at the minimum.

We simply perform some checks as we're running the createItem() function of DatabaseQueue.php. We edit both createItem() and createItemMultiple(). For createItemMultiple() we also need a way to minimize duplicates in between iterations so we also perform a check on the existing items to be processed.

Patch file attached.

EDIT: Shoot, I am dumb. I patched the purge module and not the purge_queuer_url module. I'll work on a different solution.

So I've spent some time today looking into solutions within purge_queuer_url. I was trying to tackle the problem using OP's thought process but it doesn't seem this is an easy fix with only purge_queuer_url.

It appears there are two separate causes of duplicates. One, when we trigger a cache invalidation, it will not check if there are duplicates--this is pretty simple to fix. Two, nested references, i.e. blocks, paragraphs, etc. This is a bit harder to address at the root.

We can keep track of URLs in the queue/in the process of being added to the queue but this doesn't seem to be of much help. In fact, the module already handles duplicate URLs being passed into the QueueService.php file within the purge module. In my case, when I edit a page with nested paragraphs, each entity? on the page has its own tag. Most are simply containers for other entities (references) so when we edit the actual content, this may cause multiple invalidations of the same URL because each entity on the page that gets edited counts towards a cache invalidation. Thus, if we have a paragraph that references two other paragraphs on a node, that would result in 3-4 cache invalidations.

My post #2 still works but it's not in the same module. However, purge_queuer_url and purge is maintained by the same person and the two modules seem to be heavily intertwined so that might still be a viable solution. I'm hoping the maintainer can chime in with his thoughts. Right now, purge_queuer_url handles all of the invalidations setup but then passes it to actually be invalidated to a purgeQueue service which resides in the purge module. So #2 actually works later in the process of all of this and is able to stop duplicates from entering the queue.

Please correct me if I am wrong on anything

I'm back.

So I've created a new patch. The above patch was failing on cron runs, receiving an empty array would return False which then caused an error down the road. So I've accounted for this in a new patch

Ran into the same cron error but this time for createItem(). I was initially only seeing errors from createItemMultiple() so I only fixed that method but I just added a similar check within createItem().

Just saw that this is a duplicate of https://www.drupal.org/project/purge/issues/2851893. It is indeed an issue for purge and not purge_queuer_url.

Closing as duplicate?

I'm going to reopen this as I believe the original request has merit as a work around until a more clear solution is propose for 2851893.

While 2851893 focuses on duplicate queued items - there is a valid issue with the original report of duplicate queued items in the same request, as the buffer does not deduplicate items (noted in https://www.drupal.org/project/purge/issues/2851893#comment-15182977)

Importing config is one scenario we hit where memory limits got hit after importing a very large number of config changes.

Attached is a patch that does what was originally suggested - track the purged urls of the request in a similar way to how invalidated tags are tracked.

This deduplicates items in the buffer, but if you are concerned about duplicate items in the queue (from previous/subsequent requests) you will need additional solutions such as those suggested in 2851893.

I haven't added a test with this patch but can share my local test using a similar scenario as described in the issue summary.

Setup:

1. Added 6 blocks all set to show on all pages, exported config
2. Generated 30,000 items in the traffic registry using block list and the individual block tags
3. Change the weight (in config) for the 6 blocks
4. Import config with drush -vvv cim

Without the patch:

  [debug] Calling Drush\Drupal\Commands\config\ConfigImportCommands::doImport(Drupal\Core\Config\StorageComparer) [4.89 sec, 114.07 MB]
 [notice] Synchronized configuration: update block.block.block_1. [8.12 sec, 208.87 MB]
 [notice] Synchronized configuration: update block.block.block_2. [9.19 sec, 242.77 MB]
 [notice] Synchronized configuration: update block.block.block_3. [10.21 sec, 286.69 MB]
 [notice] Synchronized configuration: update block.block.block_4. [11.28 sec, 321.07 MB]
 [notice] Synchronized configuration: update block.block.block_5. [12.38 sec, 356.42 MB]
 [notice] Synchronized configuration: update block.block.block_6. [13.48 sec, 390.32 MB]
 [notice] Finalizing configuration synchronization. [14.41 sec, 395.7 MB]
 [success] The configuration was imported successfully. [14.41 sec, 395.77 MB]

Repeated with the patch:

 [debug] Calling Drush\Drupal\Commands\config\ConfigImportCommands::doImport(Drupal\Core\Config\StorageComparer) [5.07 sec, 114.64 MB]
 [notice] Synchronized configuration: update block.block.block_1. [7.67 sec, 172.27 MB]
 [notice] Synchronized configuration: update block.block.block_2. [8.38 sec, 172.52 MB]
 [notice] Synchronized configuration: update block.block.block_3. [9.03 sec, 172.68 MB]
 [notice] Synchronized configuration: update block.block.block_4. [9.71 sec, 173.42 MB]
 [notice] Synchronized configuration: update block.block.block_5. [10.38 sec, 175.12 MB]
 [notice] Synchronized configuration: update block.block.block_6. [11.04 sec, 175.39 MB]
 [notice] Finalizing configuration synchronization. [11.96 sec, 180.77 MB]
 [success] The configuration was imported successfully. [11.96 sec, 180.83 MB]

This is only for 6 blocks - so you can imagine the imapct for much larger deployment sets.

The patch looks simple and elegant , thanks Eric.

Works as expected.