Allow Disabling of Client-side Queue Running

I fully endorse this! We are experiencing issues with purging Akamai cache. Akamai reacts on a node_update() which happens first, then you need to load the next page and hopefully let the client purge the cache. What if this never happens? Then you are stuck with the default TTL/max-age.

My path in this situation would differ though. I would like to see a config option to simply choose between client-side and server-side purging. Server-side purging will react on node_update/deletes etc. That way it still happens instantly and we can control that the Varnish purge happens consistantly and before Akamai.

Cross-linking issue I opened & closed in Akamai in favor of this solution:
#2311361: Purge after Acquia Purge (Varnish)

Cross-linking a previous issue on using Cron.

#2194387: Purge queue processing by cron

Here is a patch that:

1. Adds a "Use server-side purging for Acquia Purge" checkbox on admin/config/system/expire. I am not sure this is the best place for this but it works for the POC and appears to fit in okay there.
2. Incorporates patch #0 but I renamed the variable to "acquia_purge_enable_server_side_purging" versus "acquia_purge_enable_client_side_purging". Patch #0 effectively stops the drupal_add_js() from ever loading to the client and hitting the callback to purge the queue.
3. Adds an acquia_purge_node_update() to purge the queue on a node_save().
4. Adds an acquia_purge_node_delete() to purge the queue on a node_delete().
5. Lastly moves the Acquia Purge invocations of hook_node_update/delete to run after Expires do since Expire builds the queue to purge.

Updated #3 to invert the check for the server-side-enabled variable and to check if the requirments have been met before attempting to purge the queue in the hook_node_update/delete invocations.

Hey guys!,

Thank you for your ideas, patches and comments.

I didn't end up using the code here above because of a few reasons:

1. I decided to go with a slightly different feature set, slightly.
2. hook_node_save() and friends shouldn't see implementations in Acquia Purge, that's really the task of the Expire module I'm depending on. Adding events in them, is broken (by design) and therefore no option for me.
3. The implementation had to occur not in hook_init() but in _acquia_purge_queue_is_user_purging()

But please don't get me wrong, it did convince me we need this - combined with 2194387 - and decided to bundle it into one feature. That means that adding $conf['acquia_purge_cron'] = TRUE; to settings.php will enable cron-based processing and disable the UI processor. That also means that the client-side processor will remain enabled by default and this remains disabled for most users, as I really wan't to challenge people to think about the consequences before just enabling something. That's also the reason I'm not adding bells and whistles in the administration backend in the form of checkboxes, but, its available now.

Its committed to branch 7.x-1.x which means it will end up in release 7.x-1.0, when that release takes place is a little bit a black box to me as well, but will some luck this week or next week... quality over speed!

(7.x-1.x *)$ git commit FAQ.txt acquia_purge.admin.inc acquia_purge.drush.inc acquia_purge.module
[7.x-1.x abb2241] 2194387, 2228949: by popular demand, it is now possible to *purge from cron and without client-side UI* by setting "$conf['acquia_purge_cron'] = TRUE;" in settings.php.

That also means I'll close the ticket, as I usually do for every resolved issue that's due for release.

Thanks for the feedback. Unfortunately the cron based solution won't work for us as we need the purge done immediately after the content is edited as then Akamai gets a purge request and it needs fresh content too.

Any ideas on that?

http://cgit.drupalcode.org/acquia_purge/commit/?h=7.x-1.x&id=abb22415662...

Sorry to set this to needs review again but I was hoping to get some feedback on a better solution for those that need to purge content more quickly and also use Akamai.

Our use case is that our editors want there content to be shown as quickly as possible to the public after publishing, Akamai already imposes up to a 7 minute delay to invalidate their all of their global nodes/caches so we have to act immediately on node save. Therefore we need to first purge varnish and then immediately after purge Akamai. A cron based solution just won't work for our immediacy, unless maybe I am missing something obvious.

p.s. I also just upgraded from 7.x-1.0-beta2 to 7.x-1.1 and had to re-roll our patch for now so I also attached it.

What if we were to support a similar setting as we do for enabling cron based purging and ripped out the form settings? I could update the readme to have similar verbiage as the cron instructions. I also just realized I would need to update the reporting feature to reflect the correct proccessing mode too.

I removed the hook_node_update and hook_node_delete() operations since those seem to be unnecessary and integrated it with the changes in #2400967: Force Acquia purge to run and clear it's queue before Akamai (integration with Expire).

Using simpler variable_get() instead.

More simpler variable_get() calls.

Alright, I see partly why nielsvm didn't want a server side purge option, it is because there is a max # of items to be popped based on server/domain checks and sometimes the do/while loop that does the popping doesn't fully empty the queue. This leads the queue to fill up to quickly and then quickly reaches the AP shutdown limit. This happens quite a bit with some sites we have purging more than one domain. In the FAQ it says:

Source: FAQ.txt

The while loop does not run until the queue is empty but runs as long as PHP has resources for (and runs longer on the command line). The queue might be empty already in many cases by the end of that loop, but its definitely no guarantee and therefore having permanent processing of the queue (cron) is preferable.

You can always check upon the queue state by querying:
var_dump(_acquia_purge_queue_stats());

Its fine to do 10-50 paths in a do-while loop during a single request but whenever the user has many domains or many load balancers, you will quickly run into the necessity splitting things out using batch API. Just keep on calling _acquia_purge_queue_pop('_acquia_purge_queue_processpurge') till it returns a fully empty array.

So, using the batch API or Cron seem like the only viable option if you don't want to use the unreliable client-side purging.

Or maybe we can have a reliable server side purging option by just implementing the batch API suggestion?

Btw, thanks for all the work on this Dave Reid! Also, if we are going to have a server side purge option I like the idea of removing the form variable and just leaving it as a conf variable like the cron conf var is currently.

Just keep on calling _acquia_purge_queue_pop('_acquia_purge_queue_processpurge') till it returns a fully empty array.

Isn't this what we're doing now?

I'm honestly still not convinced about the idea of a server-side processing, the module's already quite complex.

One bit of good news is that I'm also working on the Purge module for Drupal 8 (http://www.drupal.org/project/purge) in which this way of working will be possible by simply enabling its "late-runtime processor" which can even be combined with a "cron processor". So for the future, there's a good way forward in my opinion.

Another bit of news regarding acquia_purge is that its API is changing (with deprecated wrappers) as of 7.x-1.2. The new FAQ.txt section regarding the processing code is:

 Can I use Acquia Purge to programmatically purge paths I want to be cleaned?
--------------------------------------------------------------------------------
  Yes, the module has been purely designed to purge things on Acquia Cloud and
  to do it well! Its relation with the expire module is very thin for instance,
  it receives paths and wraps those to its own publicly facing API functions.


  You can queue items for purging like this:
    $queue = _acquia_purge_qs();
    $queue->addPath('node/5?parameter');
    $queue->addPaths(array('news/section1', 'contact'));

  If you run the code above as authenticated user and you aren't using cron, it
  will trigger the UI processor for that user on the next page load. However, if
  you need to do direct processing yourself, you can use "drush ap-process",
  depend on cron or do it yourself with:
    $queue = _acquia_purge_qs();
    if ($queue->lockAcquire()) {
      $queue->process();
      $queue->lockRelease();
    }

  The ApQueueService::process() call will run as long as PHP has resources
  for and depends on Acquia Purge's internal capacity calculation. When it runs
  from the CLI, it has a higher max_execution_time and will therefore process
  much more at once, then it will when triggered through a web server.

  You can always check upon the queue state by querying:
  var_dump(_acquia_purge_qs()->stats());

I haven't shot this yet, in fact, I'm between yes and no and would like to explore the idea further for 7.x-1.2. However, I do sincerely think there's a couple of changes to be made to the idea and implementation of the patch.

1. Don't disable the AJAX-processor, lets reduce and hide it
   In many cases such as a simple page edit, a in-request call to ApQueueService::process() is likely to process the whole queue. The AJAX-processor only shows up when there is stuff to process so this would already reduce its presence in the daily life of editors. It gets better, when you revoke the purge on-screen permission from editor roles, the AJAX processor will still kick in when it needs - on every admin page the editor loads - but won't show the UI at all.
2. Call ::process() from a shutdown function triggered by acquia_purge_expire_cache(), and not for CLI
   In my opinion, the ::process() call should occur either in a hook_exit() implementation or in a registered shutdown function. We can use acquia_purge_expire_cache() as trigger event for registering a shutdown function. This way, processing only occurs once during the request and at the best point possible: when most of Drupal's already done (and potential out-of-memories/script timeouts do the least harm). Also important to wrap it inside a php_sapi_name() !== 'cli', since modules like feeds on Drush scripts can cause thousands of entity_save() calls making this toxic combination.
3. Drop the form element
   I'm not against GUI's, but I agree the form should go. I'm sticking close to keeping the module a "zero conf" utility, despite its feature flags for more advanced users.

And thanks for all your efforts and thinking guys!

I'm bringing good news:

nvmourik ~/src/acquia_purge (7.x-1.x +)$ git commit -m 'Issue #2228949 by Elijah Lynn, Dave Reid, ryandekker, nielsvm: added "late runtime" processing on the server side [OPT-IN]' --author="nielsvm <nielsvm@163285.no-reply.drupal.org>"
[7.x-1.x 1c32e00] Issue #2228949 by Elijah Lynn, Dave Reid, ryandekker, nielsvm: added "late runtime" processing on the server side [OPT-IN]
 Author: nielsvm <nielsvm@163285.no-reply.drupal.org>
 3 files changed, 55 insertions(+), 3 deletions(-)

I implemented a server-side processing mode as follows:

• Opt-in feature via $conf['acquia_purge_lateruntime'] = TRUE;.
• Hides - though does NOT disable - client side AJAX processor.
• Processes from hook_exit() (only non-CLI).
• Big queue loads that ::process() cannot finish during the trigger request, will "flow over" and trigger the AJAX processor.

The experience feels very smooth and I'm certain this is close to what you wanted, though real world experience might be bumpier, time and your experiences with it will tell. At least the documentation is crystal-clear on the - quite theoretic, but unavoidable - risks involved:

║ acquia_purge_lateruntime ║ FALSE ║
When enabled, requests in which items are
added to the queue, will also immediately 
process a chunk. Queues clear quicklier   
and the (hidden) client-side processor    
runs less. However, this mode is RISKIER  
as backend pages become slower and can    
even time-out/run out of memory!          
$conf['acquia_purge_lateruntime'] = TRUE; 

UPDATE:

[7.x-1.x 7dc03ae] $conf['acquia_purge_silentmode'] = TRUE; now visually hides the client-side processor though it CANNOT be disabled. This decouples the visual hiding from cron and late-runtime mode, which was admittingly a little messy.
 2 files changed, 21 insertions(+), 17 deletions(-)

This now means that it takes one step more to "disable" hide the client-side processor and have in-request ("server side") processing enabled:

• $conf['acquia_purge_lateruntime'] = TRUE;
• $conf['acquia_purge_silentmode'] = TRUE;

In my tests, it showed that the client-side processor almost never runs anymore when the configuration of the site is simple and the queue stays low. For instance, when doing a VBO update to hundreds of nodes, it will still trigger but silently without the administrative user noticing.

I think this is the best compromise we can get.

This is a good step in the right direction and is better than what we have now because if the purge queue overflows then it will just stop running whereas this will at least purge the Varnish side of things fully.

However, this will still cause problems for us when it overflows into the client-side processor because it will purge Varnish after it purges Akamai. We would need to have Akamai check the AP queue and not purge until after it is emptied. But I don't know if the above is manageable and it does seem like maybe this would all be much easier if we went towards the route of making both an Acquia Purge and Akamai plugin for the Purge module maybe. I thought I saw you mention that that was the ultimate goal but we were waiting on a 2.x release of Purge module.

What are your thoughts on how to make the Akamai queue purge after the Acquia Purge Varnish queue?

Did #21, 7dc03ae ever get pushed? I don't see it in the logs.

Elijah,

In my tests overflow only occurs in 5-10% of all cases, usually with mass updating of content. Also, now with $conf['acquia_purge_lateruntime'] = TRUE;, processing also happens during cron and at the very END of cron, so that means processing really does happen at during the most ideal moment.

But I don't know if the above is manageable and it does seem like maybe this would all be much easier if we went towards the route of making both an Acquia Purge and Akamai plugin for the Purge module maybe. I thought I saw you mention that that was the ultimate goal but we were waiting on a 2.x release of Purge module.

This is the most ultimate goal and ideal situation, but only expect this for Drupal 8. Its only with D8 that its architecture facilitates something that becomes this complex very well. I mean, yes, this could be done with hooks in D7 but the amount of complexity will grow out of hand and make it so hard to maintain that it becomes unsustainable. As soon as the next version of acquia_purge is released, I'm going to continue working on Purge (see http://cgit.drupalcode.org/purge/tree/?h=8.x-3.x), which really is facilitating this properly. And this reminds me, I should allow D8 "Purger" plugins to be configurable in order :-).

Being pragmatic, I think this the most we can get out of it for D7 without creating ourselves a code monster.

Elijah,

Yes, eventually 7dc03ae got pushed, after I got caught up ;).
http://cgit.drupalcode.org/acquia_purge/log/

I'm actually feeling quite confident about this solution now, as long as everyone using it tests it properly enough before exposing it to end-users to aggressively. This also means I'm closing the ticket with release 7.x-1.2 at our doorstep, any feedback and improvements can be added to a reopened version of this ticket and/or into new ones that follow up.

w00sh!

Thanks for all the discussion.

We are moving towards D8 soon so I guess that is fine. We will just make sure to test and not let it overflow for now. I will try this out soon and post back if I have any issues.

Cheers

I have been testing the latest 7.x-1.2-beta1 and the latest code causes Acquia Purge to run after Akamai, aka the wrong order.

I think this is because the actual purge call isn't being made in acquia_purge_expire_cache, it just adds URLs there. And maybe because we are using the lateruntime variable?

We added support in the Akamai module to check if Acquia Purge was enabled and make it run at the end, here is the commit - http://cgit.drupalcode.org/akamai/commit/?id=4324aed3e40d49569251da2fac7...

I have been trying to fix it but it isn't coming easy to me, any ideas? We can maybe reroll a patch to Akamai to make it work with this new way in AP.

This is deviating too much, moving the Akamai side of this story into https://www.drupal.org/node/2521530

For title-readers:
The client side processor cannot be disabled, it serves as last defence barrier. With the introduction of late runtime processing in 7.x-1.2-beta1, the queue gets processed before the client-side processor has to kick in in 90+% of all cases - effectively resulting in the same. Revoking the Purge on-screen permission will visually hide the processor as well (not disable, just silences it) or this can be enforced with $conf['acquia_purge_silentmode'] = TRUE; as well.