I have a site with relatively large traffic. We use Memcached and Varnish.

Today we deployed changes in one of our features. There are multiple components in that feature, but changes were only on fields. We were reverting it in UI and we reverted just fields component. When we triggered a revert, site went down. Our logs were saying that there were to many connections on MySQL (limit is set to 300 connections).

I checked MySQL binlogs and it looks that there was happening a lot on semaphore and cache_views DB tables. I suspect that there were problems with menu_rebuild() (similar to issues described in #1093664: Excessive menu_rebuilds caused by features.content.inc).

It seems that a lot of requests try to access page when Drupal rebuilds menu router. I was not using maintenance mode. Later I checked how Drupal acts while in that mode. It looks like it still rebuilds menu on every request, which made me think that maintenance mode does not solve this issue.

I also noticed that revert clears entire views_cache table. Since I was reverting only a field on a node, I think this shouldn't happen. Since cache_views stores info about all views this results in significant overhead.

How can I prevent this from happening? I would be very happy if you could help me to debug and solve this. Why does revert of fields flush cache_views also?

Comments

hefox’s picture

hefox commented

As went over in that issue, updating features, specially on a high traffic site, needs maintenance mode.

So you're saying in d7 that anonymous traffic is triggering menu rebuilds even while in maintenance mode? That seems like another/regression. How did you test/confirm this?

slashrsm’s picture

slashrsm commented

I confirmed this with step debugging of core during a maintenance mode request. Here is the code from menu_rebuild():


  // ....

  $transaction = db_transaction();

  try {
    list($menu, $masks) = menu_router_build();
    _menu_router_save($menu, $masks);
    _menu_navigation_links_rebuild($menu);
    // Clear the menu, page and block caches.
    menu_cache_clear_all();
    _menu_clear_page_cache();

    if (defined('MAINTENANCE_MODE')) {
      variable_set('menu_rebuild_needed', TRUE);
    }
    else {
      variable_del('menu_rebuild_needed');
    }

 // ...

This code is called on every request when menu_rebuild_needed variable says that we need to rebuild menu. Even for maintenance mode requests. As far as I understand this code Drupal builds entire menu first and checks if in maintenace mode at the very end. It then marks the menu, that was just built, obsolete and requires another menu rebuild.

It looks like not just that we build menu's cache on MM requests. We also build it over and over again, as we mark it as obsolete immediately after it was built.

slashrsm’s picture

slashrsm commented

Cross posting a related core issue: #997918: Menu gets rebuilt continually during install and update (and menu system stores entire, out-of-date theme objects) (comment #17)

mpotter’s picture

mpotter commented
Status: Active » Closed (won't fix)

Closing this for lack of activity. Please re-open this issue if you can reproduce it in the latest version.