Performance issue with V7.x-1.6

It sounds like the locking layer. Perhaps first try disabling Stampede Protection (comment it out from your settings.php). If that doesn't work, also comment out the memcache-lock.inc line.

If either works then we know where to dig further to track down the problem.

Thanks -- I hope you're enjoying! :)

Ok, it's nice to have isolated the problem so quickly. Next we need to figure out which lock. The simplest thing will probably be to enable the debug log with verbose logging (http://cgit.drupalcode.org/memcache/tree/README.txt#n506), and if you could upload (or send me directly via email to jeremy at tag1consulting.com) the log. It shouldn't contain anything sensitive unless your keys themselves are sensitive -- it doesn't log the actual cache values; it's a new feature though and I've not reviewed it closely for leaking sensitive data.

I'll try and review tomorrow, I'm dropping offline now, speaking of the toddler ...

Please be sure to re-enable stampede protection first.

Hi, I have also a big performance issue just after I updated, the MySQL server is much more used than before. Removing stampede protection seems to do the trick too.

I'll see if I can have some logs too, but not during the day as it slows the website a lot.

@deranga, thanks for the debug info. The answer is not immediately jumping out at me, but looking at the release node () for related issues to see where this regression may have come from:

1. What PECL extension are you using? (Memcache or Memcached?) #2376391: Call to undefined function lock_acquire() was related to PECL Memcached and Stampede Protection. That said, I don't see any obvious way this is going to cause a regression.
2. Perhaps this regression is from the bug fix in #2413663: Notice: Undefined index: httprl lock_release memcache -- specifically, in earlier releases we could have freed locks that another process originally grabbed (defeating the purpose of the lock), see the fix here: http://cgit.drupalcode.org/memcache/commit/?id=06e9348 -- as you're only seeing this problem on one specific page, this seems a likely cause.
3. Perhaps you've got some large items being stored/retrieved correctly now after #2415109: Notice: unserialize(): Error at offset 0 of 2527853 bytes in _dmemcache_get_pieces() (line 329 of modules/memcache/dmemcache.inc). -- we log when we split the item apart and store it, but we don't log each time we retrieve it from the cache. That said, we add "_multi" to the key name, and grepping through the log you provided I'm not seeing any keys with multi in the name.

From the above list, I think #2 is most likely the issue. Going back to your log, this does look suspicious (I've truncated the key names for clarity and to avoid leaking anything that shouldn't be public):

1489533542|add|semaphore|...cache_page|1
1489533554|get|semaphore|...cache_page|1
1489533554|delete|semaphore|...cache_page|1

This page is being rebuilt for 12 seconds -- is this expected?

And again, only 4 seconds later, this time taking 13 seconds:

1489533558|add|semaphore|...cache_page|1
1489533565|get|semaphore|...cache_page|1
1489533565|delete|semaphore|...cache_page|1

And again, only 6 seconds after that, this time taking only 7 seconds:

1489533571|add|semaphore|...cache_page|1
1489533578|get|semaphore|...cache_page|1
1489533578|delete|semaphore|...cache_page|1

And finally 5 seconds later, taking 12 seconds again:

1489533583|add|semaphore|...cache_page|1
1489533595|get|semaphore|...cache_page|1
1489533595|delete|semaphore|...cache_page|1

Why is this specific page-cache getting flushed / being rebuilt so often? Is this expected? Also, why is it taking so long to rebuild the page? (It looks like there's a lot of Views stuff happening on that page ...)

If the stampede protection is negatively impacting this page, you could explore adding to $conf['memcache_stampede_protection_ignore'] to avoid locking on these paths.

Perhaps temporarily try adding the page_cache to memcache_stampede_protection_ignore -- whether the problem does or does not go away will be an interesting datapoint. (I'd recommending cutting and pasting the entire example from README.txt into your settings.php, and then adding the page_cache table -- otherwise you could create new problems by removing other tables/keys from the ignore variable.

Doing a egrep "add|delete" on your log file, the page cache looks to me to be the only lock that's consistently being held a long time. That said, while that's being held, there's also a number of lock that look something like "MemcacheKeyPrefix-semaphorememcache_ctools_export%3Aviews_view%3ASearch%3Acache_views also being grabbed and held. If the test of disabling for page_cache doesn't work, perhaps re-enable, then try with cache_views.

Beyond that, I do see a lengthy lock held on cache_metatag once, but not consistently. (It holds it for 1-12 seconds in the log sample I have).

@deranga any updates?

I am unable to duplicate this problem -- blocked until I get more information. Updating status.

Thanks for the update!

There's now >5,000 users of this release ( https://www.drupal.org/project/usage/memcache ) and we're not seeing a surge of performance complaints, so currently I suspect there's something specific with your codebase or configuration. But, we'll see! :)

Yesterday I pushed out an update that included Memcache 7.x-1.6 and my server load has been going crazy ever since. Right now this module is the prime culprit. What do you need to know?

Did you try #2 above, and see if it helps? Maybe work through the suggestions already in the ticket if your symptoms are the same? Otherwise open a new ticket and describe your symptoms.

When I updated the site on Monday, the server went down almost immediately, prompting 502 Bad Gateway errors. We rebooted and it it came back, but over the following 24 hours or so, anytime we had even moderate traffic CPU and memory usage would spike, making the server unresponsive. This was happening every few minutes Tuesday morning. It seemed like much of this was due to communication between Apache and the memcache server. Tuesday afternoon I set $conf['memcache_stampede_protection'] = FALSE; and it seemed to calm everything down. Then we had a couple people report Bad Gateway errors yesterday, so out of an abundance of caution I rolled back to memcache-7.x-1.5.

I didn't setup the server and I don't manage it, so my familiarity with the configuration is minimal, but here's what I can tell you... We're running:

• Drupal 7.54 (it's a single codebase. No multisite)
• Ubuntu 14.04.5 LTS
• PHP 5.5.9-1ubuntu4.21
• MySQL 5.5.54-0ubuntu0.14.04.1
• Memcache v3.0.8 PECL extension
• Memcached v1.4.14

The relevant Drupal settings I found are:

$conf['cache_backends'][] = 'sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
$conf['page_cache_without_database'] = TRUE;
$conf['page_cache_invoke_hooks'] = FALSE;
$conf['memcache_servers'] = array(
  '192.168.201.220:11211' => 'default',
);
$conf['memcache_bins'] = array(
  'cache' => 'default',
);

Unfortunately there's not a lot to go on. Perhaps try collecting debug, as described in #4 above? It would also be good to confirm definitively that disabling does or does not solve the problem for you.

Old ticket, no updates in >3 years, closing