The attached patch has the goal to make the module more failure tolerant. This by ensuring that a failing cache not fully breaks the page and avoiding unnecessary slow down on failure.

Changes:

  • dmemcache_get(): Make sure the variable $result is set in any case.
  • dmemcache_object(): If a connection to a server fails don't try to connect again within the same request. Avoids that we run into the connection timeout multiple times.
  • MemCacheDrupal->clear(): Stop execution if there's no memcache connection.
  • Support from Acquia helps fund testing for Drupal Acquia logo

    Comments

    girishmuraly’s picture

    Your patch certainly reduced the number of warnings I get on screen if a particular bin (cache_menu's in my case) was killed. I still get a couple of them though:

    Notice: Memcache::connect() [memcache.connect]: Server 127.0.0.1 (tcp 11212) failed with: Connection refused (111) in dmemcache_object() (line 293 of /var/www/html/drupal/sites/all/modules/contrib/memcache/dmemcache.inc).

    In my case, line 293 is marked in the code snippet below:

    foreach ($memcache_servers as $s => $c) {
            if ($c == $cluster && (!is_array($failed_connection_cache) || !array_key_exists($s, $failed_connection_cache))) {
              list($host, $port) = explode(':', $s);
    
              // This is a server that belongs to this cluster.
              if (!class_exists('Memcached') && !$init) {
                // If using PECL memcache extension, use ->connect for first server
                if ($memcache->connect($host, $port)) { /*** LINE 293 ***/
                  $init = TRUE;
                }
              }
              else {
                if ($memcache->addServer($host, $port) && !$init) {
                  $init = TRUE;
                }
              }
              if (!$init) {
                // @TODO Find a way to create a logentry. This is to early in
                // bootstrap to use watchdog.
                $failed_connection_cache[$s] = FALSE;
              }
            }
          }
    
    pillarsdotnet’s picture

    Status: Needs review » Needs work
    --- dmemcache.inc
    +++ dmemcache.inc
    @@ -93,6 +93,7 @@ function dmemcache_add($key, $value, $exp = 0, $bin = 'cache', $mc = NULL, $flag
      */
     function dmemcache_get($key, $bin = 'cache', $mc = NULL) {
       global $_memcache_statistics;
    +  $result = NULL;
       $full_key = dmemcache_key($key, $bin);
       $statistics = array('get', $bin, $full_key);
       $success = '0';
    

    This is wrong. The function documentation says that it returns FALSE on error, which is different from NULL.

    See correct patch at #1307078: Fix Notice: Undefined variable: result in dmemcache_get() (line 114 of /var/www/sites/all/modules/memcache/dmemcache.inc).

    das-peter’s picture

    Status: Needs work » Needs review
    FileSize
    2.22 KB

    Wow, long time ago I've created this patch. Luckily I've found some time to re-roll it today.

    @girishmuraly: I'm not sure if we should suppress this error - since it's really an essential one. Goal of the patch is only to hit this error once and by that not slowing down the site build more than really necessary. The other difficulty is that this code runs early in bootstrap and thus I don't have the possibility to log it into watchdog (at least as far as I know :) )

    @pillarsdotnet: Thanks for reviewing it! I've changed that now.

    pillarsdotnet’s picture

    pillarsdotnet’s picture

    Suggestions:

    Change:

    +        if ($c == $cluster && (!is_array($failed_connection_cache) || !array_key_exists($s, $failed_connection_cache))) {
    

    To:

    +        if ($c == $cluster && !isset($failed_connection_cache[$s])) {
    

    Change:

    +        if (!$init) {
    +          // @TODO Find a way to create a logentry. This is to early in
    +          // bootstrap to use watchdog.
    +          $failed_connection_cache[$s] = FALSE;
    +        }
    

    To:

    +        if (!$init) {
    +          error_log(st('Failed to connect to memcache server: %server', array('@server' => $s)));
    +          $failed_connection_cache[$s] = FALSE;
    +        }
    
    das-peter’s picture

    @pillarsdotnet: Thank you again for the review :)

    I agree with you if ($c == $cluster && !isset($failed_connection_cache[$s])) { is way more readable.
    However to do this I had to change this $failed_connection_cache[$s] = FALSE; to this $failed_connection_cache[$s] = TRUE;. Otherwise isset / empty don't recognize that the array item exists. FALSE would be nice because the server is actually "false" - but I guess the readability has priority. Other opinions on that?

    error_log() and st() are indeed a nice idea - thanks!

    pillarsdotnet’s picture

    The functions isset() and empty() do not work the same way.

    A variable set to FALSE will be both set and empty.

    A variable set to NULL will be both unset and empty.

    The following should output true.

    php -r '$x = FALSE; var_export(empty($x));'
    

    So should this:

    php -r '$x = FALSE; var_export(isset($x));'
    

    And this:

    php -r '$x = NULL; var_export(empty($x));'
    

    But this will output false:

    php -r '$x = NULL; var_export(isset($x));'
    
    das-peter’s picture

    Wow, I really wasn't aware of that - thank you for clarifying this :)

    pillarsdotnet’s picture

    Status: Needs review » Reviewed & tested by the community

    Patch in #6 looks good to me. Using in production now.

    pillarsdotnet’s picture

    Status: Reviewed & tested by the community » Needs work

    Found one mistake, which was my own fault:

    +          error_log(st('Failed to connect to memcache server: %server', array('@server' => $s)));
    

    Should be:

    +          error_log(st('Failed to connect to memcache server: %server', array('%server' => $s)));
    

    (sorry)

    das-peter’s picture

    Status: Needs work » Needs review
    FileSize
    2.15 KB

    Oh, I missed that to *blush*
    Thanks for letting me know - patch updated.

    pillarsdotnet’s picture

    Status: Needs review » Reviewed & tested by the community

    Good to go.

    Berdir’s picture

    Subscribe, will try to review and test this asap.

    Berdir’s picture

    Status: Reviewed & tested by the community » Needs work
    +++ b/dmemcache.incundefined
    @@ -393,6 +394,10 @@ function dmemcache_object($bin = NULL, $flush = FALSE) {
    +        if (!$init) {
    +          error_log(st('Failed to connect to memcache server: %server', array('%server' => $s)));
    +          $failed_connection_cache[$s] = TRUE;
    

    Is there a specific reason to not use watchdog() here? That is imho the correct function to use for messages like this, unless there is a reason why it can't be used.

    +++ b/memcache.incundefined
    @@ -146,6 +146,10 @@ class MemCacheDrupal implements DrupalCacheInterface {
    +    if ($this->memcache === FALSE) {
    +      // No memcache connection.
    +      return;
    

    If this happens, we should have already logged a message at the time the connection failed, right?

    pillarsdotnet’s picture

    Status: Needs work » Needs review

    Is there a specific reason to not use watchdog() here? That is imho the correct function to use for messages like this, unless there is a reason why it can't be used.

    Yes. This code is invoked during bootstrap, before watchdog is available. So we have three choices:

    1. Fail silently.

    2. Fail while gracefully reporting an error using the only means available.

    3. Deliberately invoke watchdog() to produce an ungraceful fatal error.

    If this happens, we should have already logged a message at the time the connection failed, right?

    Precisely. No need to log the same error multiple times.

    das-peter’s picture

    @berdir: Thanks for the review.
    @pillarsdotnet: Thanks for answering the questions.

    You guys are awesome, your responsiveness is very much appreciated :)

    Berdir’s picture

    I see. Looking at watchdog(), which is defined in bootstrap.inc and therefore always available, it is basically a no-op if module_implements() is not defined. Problem is, in that case, the message won't be logged at all. Not sure about trigger_error() either, not sure what happens if e.g. a E_USER_WARNING is triggered during bootstrap.

    So error_log() might very well be the only option here, but maybe you could add a quick comment explaining why it's used instead of something else?

    das-peter’s picture

    Thank you for the review. Indeed trigger_error() is a good idea since it triggers the Drupal error handler.
    I've changed the code regarding your suggestions.
    Besides that I fixed a structural flaw and I also changed the memcache-lock.inc.
    Without the change the lock seems to kill the script execution, by a to high nesting level, if the memcache isn't reachable.
    Now memcache-lock.inc falls back to the default lock mechanism if memcache isn't available.

    The other idea I was thinking of is to fallback to the default cache handler if memcache isn't available. Because in my eyes this modules highest priority is to ensure the execution is not slowed down - regardless of what happens. Second priority has of course to gain a performance boost :)

    catch’s picture

    Status: Needs review » Needs work

    trigger_error() is good, but error_log() should be removed.

    Do we really need to cache the fact that connections failed?

    +// Check if memcached is available - if not switch back to default lock handler.
    +if (!dmemcache_object('semaphore')) {
    +  require_once DRUPAL_ROOT . '/includes/lock.inc';
     }
    +else {
     
    

    I'm not keen at all on doing this, conditionally defining functions is not pretty although I don't see a way around that without backporting #1225404: Modern event based lock framework proposal to D7 (which I would like to do though if it gets in).

    das-peter’s picture

    Thanks for the review.

    • error_log() removed
    • The cache avoids that memcache runs in the connection timeout again. And it avoids multiple reports about the failed connection - a screen full with that message isn't better than one.
    • I agree that the code in the lock is dead ugly - but it works. And we should definitely prefer the ugly code instead a broken page. I've added a @TODO tag to that part now.
    das-peter’s picture

    Status: Needs work » Needs review

    Oh, needs of course again a review or at least more opinions about the ugly code :D

    catch’s picture

    I'd be happier with the lock.inc change if that was moved to an include maybe. At least then there's not weird indentation if the function declarations.

    das-peter’s picture

    What about this solution:
    Moved code of memcache-lock.inc into memcache-lock-code.inc and replaced code in memcache-lock.inc with a conditional include.

    Code stays readable & maintainable. Only the files could be a little bit confusing. Nevertheless, even if someone accidentally uses memcache-lock-code.inc directly the functionality won't break.

    catch’s picture

    Looks much better - there's at least one comment over 80 chars though, also @TODO should be @todo.

    +// @TODO get rid of this conditional include as soon as this is done: http://drupal.org/node/1225404
    
    das-peter’s picture

    Thanks for the hint - changed.

    catch’s picture

    Version: 7.x-1.x-dev » 6.x-1.x-dev
    Status: Needs review » Patch (to be ported)

    OK this looks fine now. Committed/pushed to 7.x-1.x, we could probably backport this to 6.x

    • Jeremy committed 539594d on 6.x-1.x
      Issue #1184678: failure tolerance fixes for Drupal 6
      
    • Jeremy committed 5502a40 on 6.x-1.x
      Issue #1184678: enhance fault tolernace
      
    Jeremy’s picture

    Status: Fixed » Closed (fixed)

    Automatically closed - issue fixed for 2 weeks with no activity.