Hi,

We are using CloudFlare CDN (no tags-based invalidation support) and facing the following problems:

  • After saving a Drupal node, we end up with a massive queue (drush p:queue-browse lists 1880 items of URL type)
  • If we run drush p:queue-work --finish, the queue is cleared successfully but the cache doesn't look invalidated when landing of the node page (CF cache is hit and age of the page is not close to 0)

Please note:

  • Using Purge UI block invalidates the CloudFlare cache successfully for a page
  • Running drush p:invalidate url WEBSITE_URL invalidates the Cloudflare cache successfully for a page

Any ideas?

A possible workaround could be to add a trigger to invalidate the page cache programmatically after creating/saving a content.

Comments

GuillaumeG created an issue. See original summary.

guillaumeg’s picture

guillaumeg commented

Ok, ended up triggering the following in a service on hook_entity_insert() / hook_entity_update(): 

public function purgeCache(EntityInterface $entity) {
    $invalidations[] = $this->purgeInvalidationFactory->get(
      'url',
      $entity->toUrl('canonical', ['absolute' => TRUE])->toString(),
    );
    try {
      $this->purgePurgers->invalidate($this->processor, $invalidations);
    }
    catch (DiagnosticsException | LockException | CapacityException $e) {
      $this->messenger->addError($e->getMessage());
    }
  }
szy’s picture

szy commented
Version: 8.x-1.0 » 8.x-1.1-rc1

Hello Guillaume,

exactly the same happens to me. A small site, only 50 nodes - editing one of them puts all of them in the queue.

Is it necessary? Looks like wasting of resources, wasting of Cloudflare quota to me.

Szy.

szy’s picture

szy commented
Category: Support request » Bug report
Priority: Normal » Major

Now I see, that every cron run fills the queue with the same nodes - without even touching them. Now - with 50 nodes in the database - I have my queue filled with 620 nodes!

When I look into the purge queue browser I see my nodes listed few times - all of them with "new" status.

Marking it as a major bug.

Szy.

D 10.2.2, URLs queuer 8.x-1.1-rc1

szy’s picture

szy commented

No, sorry, it's not because of the cron run.

It happens with every install/uninstall operation - every time I see:

queue: urlpath: added 52 items

(all my content nodes).

Szy.

finn lewis’s picture

finn lewis
Location Oxford
commented

We have the same issue.

Ended up with a 2.2GB purge_queuer_url table. Had to uninstall the module.

Drupal 10.2.4

PHP 8.2.15

purge_queuer_url 8.x-1.0

It would be good to find time to fix this as without it enabling Cloudfront caching is kind of unsustainable.

Any suggestions from the maintainers would be most welcome.

szy’s picture

szy commented
Priority: Major » Critical
ericgsmith’s picture

ericgsmith commented

The issue of duplicate items going into the queue is a purge issue https://www.drupal.org/project/purge/issues/2851893

I have found success with Jonathan's module https://www.drupal.org/project/purge_queues mentioned in that thread.

For larger sites we still find issues with the performance as the duplicate check queries items individually.

I currently use this patch https://www.drupal.org/project/purge_queues/issues/3379798 along with his module which has been good for a year or so.

The issue of actions like saving one node and its filling up the queue with all items is down to your site and cache tags. Have a look at the cache tags stored against the urls. Common culprits are things like node_list from views. Modules like https://www.drupal.org/project/views_custom_cache_tag may help if that is the case.

As for a 2.2GB url table, that is huge! I'm sorry don't have much to offer, try analyze what is in there and if you can ignore any it through the blacklist. Things like facets and search terms can flood the table, if they are cached consider setting a lower lifetime and ignoring them. Cache invalidation for some of the site maybe be better than none.

This module also has an issue that pages never expire/ drop off the registry so crawlers and garbage urls can really pollute the table. https://www.drupal.org/project/purge_queuer_url/issues/3045503

guillaumeg’s picture

guillaumeg commented

Hi @ericgsmith,

Thanks for your detailed answer.

I tried your suggested solutions:

Unfortunately, I did not see any improvements.

I checked the X-Drupal-Cache-Tags headers and did not notice anything unusual for the nodes I saved during testing.
I found the same cache tags when browsing the SQL table purge_queuer_url_tag.

For other developers who need to know how to display Drupal Cache Tags, you can refer to https://www.drupal.org/docs/8/api/responses/cacheableresponseinterface#d...

However, I did notice that the queue was almost always equal to the Traffic Registry size, and saving the node again would simply double the size of the queue.

Example (after training the traffic registry with wget and registering 85 items):

  • Traffic Registry size: 85
  • Queue size: 0

After saving a node:

  • Traffic Registry size: 85
  • Queue size: 85

After saving the same node again:

  • Traffic Registry size: 85
  • Queue size: 170

Is there anything I'm missing here?

mozh92’s picture

mozh92 commented

Hello Guillaume,
I had the same problem with duplicate all items after resave node, I installed module https://www.drupal.org/project/purge_queues and it was resolved

ericgsmith’s picture

ericgsmith commented

Thanks for confirming @mozh92

@GuillaumeG did you change the purge config to use the new unique queue? Sounds like maybe the old queue plugin / table was still in use

I'm not really sure what to do with this issue - it sounds like a support request but maybe we need to update the docs to highlight that module + common pitfalls of this module.

ericgsmith’s picture

ericgsmith commented
Component: Code » Documentation
Category: Bug report » Support request
ericgsmith’s picture

ericgsmith commented
Priority: Critical » Normal

Also setting back down to normal priority - these issues are not specific to this module but the whole purge framework and I think reasonable approaches have been suggested / discussed here.

ericgsmith’s picture

ericgsmith commented

Hi team,

Fantastic news to share - https://blog.cloudflare.com/instant-purge-for-all/

Cloudflare now supports tag invalidation for all account levels - if you are using Cloudflare I suggest trying cache tag invalidation so that you can move away from this module.