Problem/Motivation
I can see that the purge by cache tags is piling up multiple queues of items e.g. if a client resaves the same item, or two separate items e.g. both result in node_list tag needing a clear. The queue can quickly get out of hand with duplicates particularly in a hosting like Upsun where the frequency of cron jobs is maximum once every 5 minutes.
Steps to reproduce
- Set up purge by cache tag using queue mechanism.
- Change a few items or same item multiple times.
- Run cron very seldom
- Review queue
Proposed resolution
Cache tags invalidated while auto-purge is queueing are merged into a deduplicated pending set instead of creating a queue item per invalidation, so repeated invalidations of the same tag no longer pile up duplicate queue items. Cron drains the pending set FIFO into queue items batched within Cloudflare's per-request tag limit. A new "Queued Tags" page shows what's pending/queued and lets an admin trigger a drain manually. Advisory warnings flag when the queue may be growing faster than it can be purged. Full details in the module's CHANGELOG.
Update the queue purge system to review existing queued items and combine into larger requests within Cloudflare's limits in a FIFO (first in first out) way. Cloudflare's rules on this: https://developers.cloudflare.com/cache/how-to/purge-cache/purge-by-tags/ ie
- A single HTTP response can have more than one Cache-Tag HTTP header field.
- The minimum length of a cache-tag is one byte.
- Individual tags do not have a maximum length, but the aggregate Cache-Tag HTTP header cannot exceed 16 KB after the header field name, which is approximately 1,000 unique tags. Length includes whitespace and commas but does not include the header field name.
- For cache purges, the maximum length of a cache-tag in an API call is 1,024 characters.
- The Cache-Tag HTTP header must only contain printable ASCII encoded characters.
- Spaces are not allowed in cache-tags.
- Case is not sensitive. For example, Tag1 and tag1 are considered the same.
Remaining tasks
Merge request + test coverage
User interface changes
N/A
API changes
N/A
Data model changes
N/A
Issue fork cloudflare_purge-3611366
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #3
scott_euser commentedComment #4
scott_euser commentedUpdated issue summary + changelog with the plan; its quite a big rework because the queue worker system fundamentally is problematic, particularly with tags like e.g. node_list which get queued and re-queued over and over. A dedicated table with a FIFO setup of cache tags solves this, it keeps the number of items far more manageable and the queue can be cleared much more efficiently without so many wasted API calls.
I considered whether adding to the queue should scan all pre-existing queued items and update them instead until they run out of space, but its A) far less efficient and B) potentially incompatible with non-database queue processors
Also added another page that lists the queued tags + gives some warnings/advice to site builders if the list is long depending on their configured settings.
Beyond that added update hook + test coverage.
Comment #8
flashwebcenterHello,
Thank you for this contribution. The dedicated FIFO deduplication table is a solid approach and keeps the implementation queue-backend agnostic, as you noted.
I reviewed it closely and added several hardening changes within your original design:
Verified on Drupal 11: the full unit and kernel suites, PHPCS, and PHPStan are clean. Merging shortly. Thank you for the solid groundwork.
Best wishes,
Alaa
Comment #9
flashwebcenterComment #11
scott_euser commentedGreat thank you very much!