For example, if I break my Memcache URLs like so:
$conf['memcache_servers'] = array(
'some-broken-server-url-1.net:11211' => 'default',
'some-broken-server-url-2.net:11211' => 'default',
);
Then what happens is during Drupal's bootstrap, variable_initialize() inside bootstrap.inc gets called, which runs this:
if (!lock_acquire($name, 1)) {
// Another request is building the variable cache.
// Wait, then re-run this function.
lock_wait($name);
return variable_initialize($conf);
}
So basically, if it can't acquire a lock, then it re-runs recursively. However, the lock_acquire() inside memcache-lock-code.inc doesn't seem to be able to intelligently handle the case where it's pointing at an invalid Memcache server. It looks like this:
function lock_acquire($name, $timeout = 30) {
global $locks;
// Ensure that the timeout is at least 1 sec. This is a limitation
// imposed by memcached.
$timeout = (int) max($timeout, 1);
if (dmemcache_add($name, _lock_id(), $timeout, 'semaphore')) {
$locks[$name] = _lock_id();
}
elseif (($result = dmemcache_get($name, 'semaphore')) && isset($locks[$name]) && $locks[$name] == $result) {
// Only renew the lock if we already set it and it has not expired.
dmemcache_set($name, _lock_id(), $timeout, 'semaphore');
}
else {
// Failed to acquire the lock. Unset the key from the $locks array even if
// not set, PHP 5+ allows this without error or warning.
unset($locks[$name]);
}
return isset($locks[$name]);
}
So the lock_acquire there just continues to return FALSE, which causes variable_initialize() to re-run itself forever.
Thoughts? I'm having trouble finding a way to intelligently handle this, outside of just having lock_acquire eventually throw an error if it gets called a certain number of times with the same $name and fails every time.
Comments
Comment #2
mcrittenden commentedComment #3
mcrittenden commentedComment #4
mpotter commentedI think this is related to an old issue I posted to: #1595070: Consider removing the cache in variable_initialize(). We fixed this infinite loop in Pressflow for D6 (pull request linked above) but never came up with a patch for D7.
Comment #5
jeremy commentedI can't duplicate this. If I intentionally specify all bad memcache servers (or stop the memache servers), Drupal automatically falls back to using the core lock implementation. I've been unable to get stuck in an infinite loop.
If you're still able to duplicate this, please provide details on how you have things configured so this happens.
Comment #6
jeremy commentedI'm re-opening this. The only way I can duplicate it is with PECL Memcache and by commenting out the calls to
connect()andpconnect()with stampede protection enabled, then restarting Memcached -- but this has me concerned that there's a lingering bigger problem here.Anyone else running into this problem, some questions:
Comment #7
jeremy commentedComment #8
jeremy commentedComment #9
jeremy commentedI've committed some logic to prevent this infinite recursion, as part of #332094: Add support for server weight.