Problem/Motivation

I can see that the purge by cache tags is piling up multiple queues of items e.g. if a client resaves the same item, or two separate items e.g. both result in node_list tag needing a clear. The queue can quickly get out of hand with duplicates particularly in a hosting like Upsun where the frequency of cron jobs is maximum once every 5 minutes.

Steps to reproduce

  1. Set up purge by cache tag using queue mechanism.
  2. Change a few items or same item multiple times.
  3. Run cron very seldom
  4. Review queue

Proposed resolution

Cache tags invalidated while auto-purge is queueing are merged into a deduplicated pending set instead of creating a queue item per invalidation, so repeated invalidations of the same tag no longer pile up duplicate queue items. Cron drains the pending set FIFO into queue items batched within Cloudflare's per-request tag limit. A new "Queued Tags" page shows what's pending/queued and lets an admin trigger a drain manually. Advisory warnings flag when the queue may be growing faster than it can be purged. Full details in the module's CHANGELOG.

Update the queue purge system to review existing queued items and combine into larger requests within Cloudflare's limits in a FIFO (first in first out) way. Cloudflare's rules on this: https://developers.cloudflare.com/cache/how-to/purge-cache/purge-by-tags/ ie

  • A single HTTP response can have more than one Cache-Tag HTTP header field.
  • The minimum length of a cache-tag is one byte.
  • Individual tags do not have a maximum length, but the aggregate Cache-Tag HTTP header cannot exceed 16 KB after the header field name, which is approximately 1,000 unique tags. Length includes whitespace and commas but does not include the header field name.
  • For cache purges, the maximum length of a cache-tag in an API call is 1,024 characters.
  • The Cache-Tag HTTP header must only contain printable ASCII encoded characters.
  • Spaces are not allowed in cache-tags.
  • Case is not sensitive. For example, Tag1 and tag1 are considered the same.

Remaining tasks

Merge request + test coverage

User interface changes

N/A

API changes

N/A

Data model changes

N/A

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

scott_euser created an issue. See original summary.

scott_euser’s picture

Issue summary: View changes
scott_euser’s picture

Assigned: scott_euser » Unassigned
Status: Active » Needs review

Updated issue summary + changelog with the plan; its quite a big rework because the queue worker system fundamentally is problematic, particularly with tags like e.g. node_list which get queued and re-queued over and over. A dedicated table with a FIFO setup of cache tags solves this, it keeps the number of items far more manageable and the queue can be cleared much more efficiently without so many wasted API calls.

I considered whether adding to the queue should scan all pre-existing queued items and update them instead until they run out of space, but its A) far less efficient and B) potentially incompatible with non-database queue processors

Also added another page that lists the queued tags + gives some warnings/advice to site builders if the list is long depending on their configured settings.

Beyond that added update hook + test coverage.

flashwebcenter made their first commit to this issue’s fork.

  • flashwebcenter committed 9c4ea36b on 3.1.x
    Issue #3611366: Harden pending-tags dedup/drain (data-loss, PG safety,...

  • flashwebcenter committed 9c4ea36b on 3.1.x-dev
    Issue #3611366: Harden pending-tags dedup/drain (data-loss, PG safety,...
flashwebcenter’s picture

Hello,

Thank you for this contribution. The dedicated FIFO deduplication table is a solid approach and keeps the implementation queue-backend agnostic, as you noted.

I reviewed it closely and added several hardening changes within your original design:

  • Pending rows are deleted only after queue items are stored successfully, with safe retry handling.
  • Pending tags use an upsert, and last_invalidated protects fresh invalidations from TTL expiry.
  • Purge Everything uses a state watermark instead of deleting the queue, avoiding mid-purge races.
  • Cron draining, configuration transitions, permissions, menu integration, and regression coverage were also tightened.

Verified on Drupal 11: the full unit and kernel suites, PHPCS, and PHPStan are clean. Merging shortly. Thank you for the solid groundwork.

Best wishes,
Alaa

flashwebcenter’s picture

Status: Needs review » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

scott_euser’s picture

Great thank you very much!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.