Issue Summary

When saving a node with the Purge module enabled, our content is always a version behind. In essence, we are forced to save the node changes twice in order for the updates to be seen live. The form editor also holds the old copy after the first save.

Investigation found this link describing the issue accurately: http://janezurevc.name/strange-behaviour-race-condition-during-nodesave

We have multiple servers in the mix:

1. MySQL/Memcached
2. Varnish
3. App Server

Memcached continues to hold the old copy of the edits with the Purge module enabled. Varnish cannot retrieve the updated copy because Drupal never serves it, still believing to most recent copy is the one that lives in memcache. The database does hold the newest version, however.

A restart of memcached results in Drupal serving the most recent save. Disabling the Purge module resumes normal functionality. We have resorted to using an in-house form to manually purge URLs from Varnish after a node update.

I understand that the race condition is fixed in Drupal 8 and we are waiting on a backport to Drupal 7. However, is there anything I can do that allows Purge to execute after the db transaction is complete? Since Purge uses the expire_cache hook, it might be tedious and/or difficult to implement.

CommentFileSizeAuthor
#11 expire-use_hook_post_action-2412517-11.patch3.19 KBAnonymous (not verified)
#8 expire-reset_entity_caches-2412517-8.patch2.77 KBsgdev

Comments

fewdea’s picture

fewdea commented
Issue summary: View changes

Workaround

Download and install the Hook Post Action module.

Modify the Expire module code to implement hook_node_postupdate, hook_node_postinsert and hook_node_postdelete instead of the defaults.

Modify the following lines in expire.module:

function expire_node_insert($node) {
function expire_node_update($node) {
function expire_node_delete($node) {

To look like:

function expire_node_postinsert($node) {
function expire_node_postupdate($node) {
function expire_node_postdelete($node) {

The only hooks we needed to modify were the node hooks. The Expire module has and uses many more than that, and Hook Post Action only adds node and entity hooks, but not comment or user hooks. If you were to need more hooks, you'd have to edit the Hook Post Action module to create additional hooks.

djbobbydrake’s picture

djbobbydrake commented

I can confirm that I was having the same issues with the race condition. Same architecture with Mysql, Memcache, Varnish. Implementing this solved the problem for us. Thanks!

SqyD’s picture

SqyD commented
Project: Purge » Cache Expiration
Version: 7.x-1.6 » 7.x-2.x-dev

As suggested I would see this as something that would need addressing in the Expire module as that's the component in control over what happens when. Purge will likely be just as affected as other implementors of Expires hooks such as Varnish module etc. I'm moving this issue to the Expire project.

spleshka’s picture

spleshka
he/him
Primary language English
Location UAE
commented

I am completely opened to solve this issue in the Expire module. I just want to get more details about why that race condition happens? Does varnish read a page from memcached or from app stack? Please, provide more details about integrations in your server stack.

mgzrobles’s picture

mgzrobles commented

Hi!
I think I have the same issue.
Problem:
- Edit node or nodequeu in my case.
- Expire get all urls to clear
- Call varnish to expire them
- Then a user make a request and
-- Get a Varnish MISS (correct)
-- Get a DRUPAL HIT from memcache in my case (no correct)

why can select Expire external and internal at same time?

joinso’s picture

joinso commented

Same problem here!

My config:

Cache Expiration (7.x-2.x-dev)
MemCache (7.x-1.5)
Recacher (7.x-1.0)
HTTP Parallel Request Library (7.x-1.14+50-dev)

When I save the node, Expires show the correct output: expires node/NID, url, home page...

Recacher shows sucessful load of urls.

However MemCache still serving the old page.

Any idea?

Regards,
JOINSO

zerolab’s picture

zerolab commented

Here's a a code snippet that does the above for nodes:

<?php

/**
 * Implements hook_node_insert().
 */
function my_module_node_insert($node) {
  drupal_register_shutdown_function('_my_module_post_nodehook', $node, 'insert');
}

/**
 * Implements hook_node_update().
 */
function my_module_node_update($node) {
  drupal_register_shutdown_function('_my_module_post_nodehook', $node, 'update');
}

/**
 * Implements hook_node_delete().
 */
function my_module_node_delete($node) {
  drupal_register_shutdown_function('_my_module_post_nodehook', $node, 'delete');
}

/**
 * Expires caches post node operations.
 *
 * @param  object $node
 *   The node object.
 * @param  string $action
 *   The node operation.
 */
function _my_module_post_nodehook($node, $action) {
  $modes = array(
    'insert' => EXPIRE_NODE_INSERT,
    'update' => EXPIRE_NODE_UPDATE,
    'delete' => EXPIRE_NODE_DELETE,
  );

  if (!isset($modes[$action])) {
    return;
  }

  expire_execute_expiration('node', $node, $modes[$action]);

  // Memcache uses shorthand keys (node/NID).
  // When absolute URLs are used, we need to manually force that clear.
  if (variable_get('expire_include_base_url', EXPIRE_INCLUDE_BASE_URL)) {
    cache_clear_all('node/' . $node->nid, 'cache_page', FALSE);
  }
}

Spleshka, are you happy with the drupal_register_shutdown_function() approach? If yes, will work on a patch to bring this in for Expire without the need for additional modules.

edit: added manual cache clear when expire_include_base_url is enabled.

sgdev’s picture

sgdev commented
Title: Varnish / Memcache node save race condition » Entity cache must be reset prior to expiring
Priority: Normal » Major
Status: Active » Needs review
Related issues: +#221081: Entity cache out of sync and inconsistent when saving/deleting entities
StatusFileSize
new expire-reset_entity_caches-2412517-8.patch2.77 KB

I think we may have found the solution, and I believe this is the core issue: https://www.drupal.org/node/221081

If you look at the save and delete functions for node, user, and file entities, the hooks are processed *before* running resetCache. Because the Expire module uses hook_insert, hook_update, and hook_delete, it can lead to unexpected caching results. This explains why the workaround in #1 would work.

The patch in #221081 was never backported to D7, so it is necessary to execute the resetCache immediately prior to running expire_execute_expiration calls.

Please review the attached patch, thanks.

sgdev’s picture

sgdev commented
Related issues: +#1679344: Race condition in node_save() when not using DB for cache_field, +#2470063: Does not handle Race condition

After having done more research and run patch #8 the last couple of weeks, I've decided to remove the patch I posted. It does not solve the issue, and the suggestions in #7 are a better basis for a possible patch.

Regarding the link in the original post about a race condition, I think this is probably the core of our problem. I'd recommend reviewing these posts:

* Race condition in node_save() when not using DB for cache_field: https://www.drupal.org/node/1679344
* Does not handle Race condition: https://www.drupal.org/project/memcache_storage/issues/2470063

Might want to try implementing the Cache Consistent module (https://www.drupal.org/project/cache_consistent/) and see if it helps anyone's issues. The Drupal.org site had to implement it to resolve this problem.

sgdev’s picture

sgdev commented
Title: Entity cache must be reset prior to expiring » Node save finishes after cache clear (Varnish/Memcache/Redis; race condition)
Status: Needs review » Active

Updating issue name to be more in line with the original title.

Anonymous’s picture

Anonymous (not verified) commented
StatusFileSize
new expire-use_hook_post_action-2412517-11.patch3.19 KB

Made a patch that makes use of hook post action module. It doesn't solve all use-cases (menu links are not entities f.ex.) but for nodes, users and other entities it works.