I'll be totally honest, I'm not sure if this is a documentation issue, a code issue or a non-issue.

We enabled memcache locking and stampede protection on a fairly high traffic site and noticed afterwards that the average page response increased in length by about 40%. After investigating, we found the issue was that some page requests were taking upwards of 15-25 seconds to load. Profiling those requests we discovered the bulk of time was spent with lock_wait, and always as a result of drupal_lookup_path calling cache_get(). However, cache_set() doesn't actually happen until drupal_path_footer() calls drupal_cache_system_paths().

This delayed cache_set issue is discussed here: https://www.drupal.org/node/2377587, and a fix was committed to allow specific bins to not be included. We added cache_path to the list of excluded items and the performance issue was resolved.

Should cache_path be added into documentation as an example to be excluded from stampede protection? Has anyone else encountered this issue? Is there possibly a better way to handle stampede protection in the instance of a delayed cache set?

Comments

Jeremy’s picture

Jeremy CreditAttribution: Jeremy commented
Related issues: +#2377587: Allow to disable memcache stampede protection for specific cache bins and CIDs, +#2381829: Ignore stampede protection ignoring for the theme registry and schema as well as rules.
Jeremy’s picture

Jeremy CreditAttribution: Jeremy commented
Priority: Minor » Normal
Issue tags: +7.x-1.6-blocker

Agreed, this should be auto-disabled by default. Tagging to be sure it's included in the next release.

arknoll’s picture

arknoll CreditAttribution: arknoll commented

FWIW, after excluding cache_path I am seeing a similar issue with cache_views. It appears to be similar to what is described here: https://www.drupal.org/node/1544086

Fabianx’s picture

Fabianx CreditAttribution: Fabianx commented

Should also add lookup_cache for the cache_bootstrap bin as 'registry' is also written at the end of request.

Fabianx’s picture

Fabianx CreditAttribution: Fabianx commented

To the issue itself:

Yes, theoretically memcache could provide an API to unlock a cache entry, but in general this would mean they all would need to add.

e.g.

memcache_cache_release()

However given how much needs to change and that this effectively would also disable memcache stampede protection, I don't think it is useful.

Jeremy’s picture

Jeremy CreditAttribution: Jeremy commented
Status: Active » Fixed

Added the entire cache_path bin, and lookup_cache within the cache_bootstrap bin.

I didn't see a delayed set in the views code, so I didn't add cache_views.

Committed:
http://cgit.drupalcode.org/memcache/commit/?id=05c5f37

  • Jeremy committed 05c5f37 on 7.x-1.x 
    Issue #2419757 by Jeremy, Fabianx, steel-track: Stampede Protection can...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

craigandrews’s picture

craigandrews CreditAttribution: craigandrews as a volunteer commented

This issue is not fixed in 7.x-1.5. It will be included in 7.x-1.6 (whenever that release is made).