Due to #1679344: Race condition in node_save() when not using DB for cache_field, Drupal.org needed to move the field cache to the DB. Our DB servers have been running into deadlocks which occasionally take the site down for up to ~5 minutes at time, and sometimes leads to replication breaking.

The current recommendation for Drupal.org is https://www.drupal.org/project/cache_consistent with Method #1 (alternate)

Our DB isolation level is read-committed, so method #1 is the one to use. Fabianx reviewed the code and liked that ConsistentCacheLookup had the buffer in its own class.

Deployment

Done in BZR:

  • Upgrade memcache from 7.x-1.0+5-dev to 7.x-1.3-rc1
  • Add cache_consistent 7.x-1.1
  • Apply core patch

Update settings.local.php and test on git7staging.

Deploy to production, enable cache_consistent, and update settings.local.php on both util & www*.

Comments

drumm’s picture

Rudy and I tentatively plan to deploy this around 11am US Pacific time tomorrow, in 22 hours. Downtime is not expected for deployment, but is possible if we need to roll back.

drumm’s picture

Assigned: Unassigned » drumm
drumm’s picture

Issue summary: View changes

I went ahead and reviewed the latest updates to the memcache module. They look good and straightforward. The new stampede protection looks like something that will help us as well.

I've committed all the code to BZR, since git7site and stagingwww are our only real options for testing, and we don't have other upcoming deployments.

drumm’s picture

This is deployed to git7site, and passing BDD tests well enough. (The test failures are related to not having a staging cgit setup, as usual.)

drumm’s picture

Issue summary: View changes
drumm’s picture

Issue tags: +needs drupal.org deployment

We're moving forward with this deployment.

drumm’s picture

Status: Active » Fixed

Deployed!

drumm’s picture

Issue tags: -needs drupal.org deployment
gielfeldt’s picture

Sorry to post in a closed ticket, but I'm just curious to know if everything went as planned, or if you experienced any problems.

drumm’s picture

So far it has been good. I haven't heard about any symptoms of cache inconsistency. What we had before was:

  • Issue queue listings being outdated, sometimes.
  • Sidebar on issue pages being outdated, sometimes.
  • Release packaging sometimes not showing up, it is a node_save() that adds data to file fields.

We've still been avoiding cache clears, the initial cache filling was okay.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Component: Webserver » Servers