Add locking to cache rebuild to avoid stampeding

Thanks. This looks quite ready, here a few remarks:

1. +++ b/rules.module
   @@ -318,22 +318,46 @@ function &rules_get_cache($cid = 'data') {
   +        rules_event_invocation_enabled(FALSE);
   +        watchdog('rules', 'Cache rebuild lock hit: !cid', array('!cid' => $cid), WATCHDOG_WARNING);
   +        rules_event_invocation_enabled(TRUE);
   

   This seems sane to do for all of the rules logging stuff. Maybe, we could add add a helper doing that sort of watchdog logging to the rules logger?

2. +++ b/rules.module
   @@ -318,22 +318,46 @@ function &rules_get_cache($cid = 'data') {
   +      if (isset($locking)) {
   

   How can this be false?

3. +++ b/rules.module
   @@ -1634,3 +1658,21 @@ function rules_entity_metadata_wrapper_all_properties_callback(EntityMetadataWra
   +  if (!is_null($enable)) {
   

   Minor: Should use isset() as usually.

oh, you got me wrong on 1), sry for being unspecific there. I was thinking watchdog is called from RulesLog, but it isn't. I don't think one can replace the rules logger right now, so no need to wrap the log() method call also.

I've updated the patch accordingly, please take a look. The rest of the patch looks good to me now, thanks!

Thanks, committed it!

hm, not sure what I messed up, probably I forgot to push. Committed & pushed it now.

I just wanted to ask if it is a normal behaviour if i get this regularly, i would say 3 times per hour.

I have the same question as kitikonti which I had posted here:
https://www.drupal.org/node/2406863

I have 10 rules active on my site:
- 7 act on before saving a certain content type (each one for a different content type)
- 2 are notifications when scheduler published or unpublished content
- 1 notifies a user after a successful feed import every week

There is hardly any content being saved right now, scheduler has not fired a single time and the import has not run. This does not sound like "extensive use of rules" to me. When is a cache rebuild triggered? I looks strange to me that I would get these warnings basically every few hours.

In my case, I am running Drupal Commerce. And with the site running in maintenance mode, this is triggering every few minutes. In some cases, several times in a minute.

In my case I have ~20% of db_logs containing "Cache rebuild lock hit". Is this normal? I have a pretty big site, ~10k users, Drupal Commerce, Userpoints, loads of nodes.

If this is normal, then maybe this whole preventing shouldn't be logged every single time? Cause, one full page of this log (in one minute) is pretty messy. Please give more info on that topic.

Since the recent upgrade to Rules 7.x-2.9 we are seeing "Cache rebuild lock hit: event_init" over and over again although we have nothing scheduled. Have rebuilt Rule's cache via Advanced tab several times. Drupal 7.34.

Since we upgraded to 7x.2.9 are are seeing "Cache rebuild lock hit: rules_event_whitelist" in the logs

After updating some code, with Rules 7x.2.9, our database had overrides, which I reverted. I got this message over and over:

WD rules: Cache rebuild lock hit: data [warning]

I tried to clear the cache and got a zillion of these:

WD rules: Cache rebuild lock hit: rules_event_whitelist [warning]

Drupal was hung.

#2406863: stampede protection should be flexible ("Cache rebuild lock hit" watchdog message) has now a patch to remove the spamy logging altogether. Please review and test there!

This is causing a lock-stampede on one of my sites (i.e. all mysql processes waiting on the lock to be released). Removing the lock-code brought that site back to normal again.

Turns out in my case the cache_rules table was somehow damaged (it was visible in mysql but not accessible). What happens if an exception is thrown while a process holds a lock? Is there any situation where we fail to release it?

@znerol: About the cache_rules table becoming corrupted, see #2160645: PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'cache_rules' doesn't exist.

Rules threw this error "Cache rebuild lock hit: rules_event_whitelist" three times in the same minute, and failed to execute the actions it was supposed to do.

Guess my comment belongs in a new issue.

Has this issue been fixed yet? Has anyone tried to downgrade the module version?
This is still a problem
https://www.drupal.org/node/2406863