Summary: I think there is bug in Drupal core where an old lock could exist and nothing is capable of releasing it or cleaning it out. The whole site becomes unusable and causes PHP timeouts across the entire site.
When saving ANY view on our site (event just changing a display title), we were getting a PHP max_execution error and the site would go down and never recover, a database restore is the only thing that would fix it.
We eventually tracked it back to menu_rebuild() in the views save function and found our database had an old lock in the semaphore table. It turned out this needed to be cleared in order to not crash the site when a view is saved. The expire date indicates its almost 3 weeks old.
menu_rebuild 180253639859a95a945ea260.24529215 1504271029.1801
With this old lock in the database I tried a few things (besides a sql delete statement) to see if I could get Drupal to clean it up.
Running
drush php-eval "var_dump(lock_may_be_available('menu_rebuild'));"
Results in a Boolean(false) because of the condition ->condition('value', $lock['value'])
Running drush php-eval "lock_release('menu_rebuild');" also does nothing since it also has a ->condition('value', _lock_id())
Running drush php-eval "lock_release_all('menu_rebuild');" also again does nothing since it has a ->condition('value', $lock_id)
There does not appear to be anything able to clean up an old lock. Cron runs regularly and again does not seem to clean up old expired locks. I believe I have this old lock as a result of the lock_release() and lock_release_all() having db_delete('semaphore') statements which are not being checked for success. I thought every database call should be checked for success, it seems weird these don't return a "true" when called.
I also don't really understand why there is an "expire" on locks, if nothing is cleaning up old locks from weeks ago. Seems like cron should do something with those?
I am mostly a front-end developer, so hopefully, someone more familiar with this can look into these issues.
The fix for our issue was to manually delete the old lock with SQL, and we could then safely save views again. But I am worried this will come back again and this error brings down the whole site.
Server Environment:
Two Load Balanced IIS servers running Drupal 7.52 on PHP 5.6.31 using MsSQL databases.
Comments