[support] Consequences of setting memcache_stampede_wait_time to 30?

Just gonna use this thread as some rubber duck debugging:

So I see the default for stampede_semaphore is 15 seconds in the vget(), but the lock_acquire() function $timeout default is 30 if that doesn't get set. So it does appear to be safe on some level if both core and memcache have the lock_acquire($name, $timeout = 30).

I am gathering that means it could take up to 30 seconds to acquire a lock and finally give up and I should watch for that. Given that the memcache default is 15 I wonder if we should try setting the stampede_wait_time to 15 seconds. I should add that we tried 5 seconds first but still had a high stampede after that.

Here are the relevant bits from readme.txt:

Only change the following stampede protection tunables if you're sure you know
what you're doing, which requires first reading the memcache.inc code.

The value passed to lock_acquire, defaults to '15':
  $conf['memcache_stampede_semaphore'] = 15;

The value passed to lock_wait, defaults to 5:
  $conf['memcache_stampede_wait_time'] = 5;

The maximum number of calls to lock_wait() due to stampede protection during a
single request, defaults to 3:
  $conf['memcache_stampede_wait_limit'] = 3;

When adjusting these variables, be aware that:
 - there is unlikely to be a good use case for setting wait_time higher
   than stampede_semaphore;
 - wait_time * wait_limit is designed to default to a number less than
   standard web server timeouts (i.e. 15 seconds vs. apache's default of
   30 seconds).

I see Drupal core's lock_wait and lock_acquire defaults are both 30 seconds. So I assume that this could be semi-safe to do. It just means we would need to fiddle with the Apache and AWS ELB timeouts to make sure they are both 90 seconds since we would be at 3 attempts * 30 seconds.

Our testing went well with 15 seconds so I feel safe with just memcache_stampede_wait_time at 15 seconds and leaving the rest alone, our ELB is already at 60 seconds so we are good there.

Hopefully this helps others. Feel free to dump comments here that may be relevant to others in this situation.