When executing a script or cron through drush, the late runtime processor never executes.

The cause for this seems to lie in the fact that the late runtime processor only listens to the FINISH_REQUEST kernel event. This event, however, is never executed in a console (such as drush) context.

Is there a specific reason for this? If not, a possible solution would be to also add a listener to Symfony's console.terminate terminate event.

Comments

rp7 created an issue. See original summary.

rp7’s picture

Issue summary: View changes
Status: Active » Needs review
StatusFileSize
new1.75 KB

After some more research, I came to conclusion that the kernel.terminate event might be a better fit, since its called in all contexts.

Patch attached executes the late runtime processor when the kernel is being terminate. No visible issues so far (except that the visitor gets to see the response faster :) ).

rp7’s picture

StatusFileSize
new2.11 KB

With the patch in #2 we were seeing successful invalidations not being deleted from the queue. Appears the deletion of successful invalidations was being done before the actual invalidation. Fixed it & commented the code accordingly.

FleetAdmiralButter’s picture

+1 on this. Running the late runtime processor during the TERMINATE event instead of the FINISH_REQUEST event would also avoid unnecessarily slowing down responses from Drupal. We've seen timeouts on sites with a large backlog in the purge queue.

japerry’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: +purge-3.1

Looks good to me too. Committed to the 3.1 branch.

  • nielsvm committed 0ae91b9 on 8.x-3.x authored by rp7
    Issue #3163002: Make late runtime processor compatible with drush
    
nielsvm’s picture

Status: Reviewed & tested by the community » Fixed

Thank you for this reasonable improvement, I've committed your patch and released it:

https://www.drupal.org/project/purge/issues/3236847

moshe weitzman’s picture

This will be quite a surprise for Drush users. I dont think it is expected that when you run an innocent command like drush status that a purge can be issued. I recommend rolling this back.

rp7’s picture

This will be quite a surprise for Drush users. I dont think it is expected that when you run an innocent command like drush status that a purge can be issued. I recommend rolling this back.

Could you elaborate a little bit what kind of issues this can cause? A purge can be issued as well when visiting, for example, the status page through your browser. I'm having a hard time to see the difference?

moshe weitzman’s picture

The cli is different than the web server. It is used for scripting auto action tasks where no human is present. There is no way arbitrary purges should be happening in these operations imo.

moshe weitzman’s picture

We already have drush commands for purge. Just run those If you want to purge.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

marcoscano’s picture

I wholeheartedly agree with #8 and #10 , the expectation IMHO is that a CLI execution should NOT trigger page-related events and process purger queues.

At the very minimum, this breaks sites using Config Split: #3264141: [REGRESSION] Purge 8.x-3.1+ incompatible with Config Split

rp7’s picture

The cli is different than the web server. It is used for scripting auto action tasks where no human is present. There is no way arbitrary purges should be happening in these operations imo.

I guess it depends on the task? If you have a script that changes content (or a migration even?), and you want to prevent visitors from seeing stale content, you might very well want to?

FWIW there are other modules doing pretty much the same, Search API being one of them (https://git.drupalcode.org/project/search_api/-/blob/8.x-1.x/src/Utility...).

Don't get me wrong though, I can see valid points for both sides.
In hindsight, maybe we should have made this configurable at the very least.

... the expectation IMHO is that a CLI execution should NOT trigger page-related events ...

As pointed at in my comment @ #3264141: [REGRESSION] Purge 8.x-3.1+ incompatible with Config Split, it's not the Purge module causing the KernelEvents::TERMINATE to be triggered, it's drush.

richardbporter’s picture

This change also seems to result in diagnostic errors like the following printing to the console quite a bit with various Drush commands:

[error]  diagnostics: ERROR: Purgers: There is no purger loaded which means that you need a module enabled to provide a purger plugin to clear your external cache or CDN. 

An example of when this can happen is syncing a site from another environment and then importing config which disables one purger and enables another (acquia_purge, varnish_purger).

richardbporter’s picture

One thing I don't quite understand is why it was necessary to do this when the module purge_processor_cron already exists. If you want to purge through Drush, can't you just enable purge_processor_cron and schedule drush cron calls regularly?

rp7’s picture

One thing I don't quite understand is why it was necessary to do this when the module purge_processor_cron already exists. If you want to purge through Drush, can't you just enable purge_processor_cron and schedule drush cron calls regularly?

They serve 2 completely different purposes. purge_processor_cron is interval based (every 15 minutes, for example). This ticket is about the late runtime processor, which is (nearly) instantaneous purging.

richardbporter’s picture

They serve 2 completely different purposes.

I think thats what was highlighted in https://www.drupal.org/project/purge/issues/3163002#comment-14246543. It seems like purge_processor_lateruntime was for web requests and purge_processor_cron was, indirectly, for the CLI via drush cron. At least they provided that option.

With this change, purge_processor_cron and the Drush commands specifically for purge have become somewhat irrelevant if purges will happen on any Drush command whatsoever.

rp7’s picture

It seems like purge_processor_lateruntime was for web requests and purge_processor_cron was, indirectly, for the CLI via drush cron.

The code nor documentation has any indiciation that the purge_processor_lateruntime was only meant for "web"-like requests. This is just an assumption of someone you happen to agree with.

As already mentioned in #14, I see valid points for both sides. Nevertheless, you are pointing to an "issue" that is not specific to Purge. There are probably a myriad of modules, such as Search API, which are doing "web"-like things in the Kernel::TERMINATE event. If this is incorrect (it may very well be, I don't know), either the use of Kernel::TERMINATE in all those modules should then be only limited to non-CLI requests (eg. if (PHP_SAPI !== 'cli') { ... }) - and it should be documented somewhere - or the event shouldn't be triggered by drush at all (which is something you won't be able to tackle from this issue queue).

kasperg’s picture

In case someone is interested I have added support for configuring whether to disable the late runtime processor when running CLI commands in #3163002: Make late runtime processor compatible with drush.

achap’s picture

I'm getting "MySQL server has gone away errors" after long running drush operations like search api indexing and running migrations with the lateruntime processor enabled. Those operations can put thousands of cache tags in the queue and it seems like it is trying to process many of them at the end of the request, or at the end of batches which is resulting in a packet size greater than max_allowed_packet. My max_allowed_packet setting is already quite high and I don't have much room to increase so I agree with the other comments that it would be better to at least have the option to disable it for CLI. Thanks for the MR @Kasperg

idebr’s picture

The issue referenced in #20 is this issue. The correct issue is #3462078: Make late runtime processor configurable for CLI commands