Problem/Motivation
The PECL memcached extension allows the automatic removal of dead servers. This feature could allows this module to relocated lost keys to the remaining cache servers, hence enabling high availability and resilience without any specific setup. At the moment, a single missing server equals no cache at all (but no White Screen Of Death as with Redis).
See: https://holmeshe.me/understanding-memcached-source-code-X-consistent-has... & https://www.php.net/manual/en/memcached.addservers.php
Steps to reproduce
- Install memcached php extension
- Configure this module as usual, enable memcache_admin sub module
- Set two memcached servers
- Add this config :
$settings['memcache']['servers'] = ["memcached:11211" => "default", "memcached2:11211" => "default"]; $settings['memcache']['options'][Memcached::OPT_SERVER_FAILURE_LIMIT] = 2; $settings['memcache']['options'][Memcached::OPT_REMOVE_FAILED_SERVERS] = true; $settings['memcache']['options'][Memcached::OPT_CONNECT_TIMEOUT] = 10; - Test a web page response time and check stats from /admin/reports/memcache
- Stop one of the memcached server
- Test again the web page response time and visits stats from /admin/reports/memcache
The performance are terrible and the stats page display an error.
Proposed resolution
Have a main maintainer confirm my diagnostic and determine if a fix could be made without rewriting the whole module or losing the memcache PECL compatibilty.
So far, my first approach was to "xdebug it" and catch each errors related to missing server.
Here's my first thoughts:
- Remove dependencies form
$settings['memcache']['servers'], get the servers list from
Memcached::getServerList - Check if PECL memcache allows the same behavior
Remaining tasks
API changes
Some new methods should be created.
Issue fork memcache-3462456
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
o'briatComment #3
morla commented+1 this would be such a useful addition!
Comment #4
o'briatIt seems to be supported (and now documented).
So a perf test should be enough to close this issue.
Comment #7
japerryYah, a perf test actually reported some interesting findings!
Basically memcache (module) opens a fresh, non-persistent Memcached object on every request. libmemcached counts one failure per operation and refuses to retry a server before the retry timeout, which is at least one second. So a failure limit of 2 can never be reached inside a request, and the dead server is never removed.
Soo I added an MR that makes some changes, as well as reports (properly) when a server is not there.
Comment #8
o'briatI'll try you patch asap.
The persistency problem should to be fixed in this issue: #3058121
Comment #10
japerryagreed, and that issue has also been fixed.
Comment #12
o'briatI apply the patch manually (do not apply via composer): the problem is still present.
I setup a ddev with 3 memcached, as soon as I shutdown one, response time goes from 100ms to over 6s.
The status page (/admin/reports/memcache) shows 3 setuped servers and only two available ones.
Drupal 11.4.7 , PHP 8.4.24, PECL Driver in Use: Memcached v3.4.0
Here's my settings :
Comment #13
o'briatIf combined with persistency patch (#3058121), I got the following errors on status page :
Failed to connect to server at memcached1:11211.
Failed to connect to server at memcached2:11211.
Failed to connect to server at memcached3:11211.
There may be a problem with your Memcache configuration. Please review README.txt for more information.
Comment #14
japerryInteresting! I attempted to reproduce your setup, with 3 memcached containers on a ddev setup following the README, with one container stopped.
What happens is not the connect timeout. Stopping a container removes its hostname from Docker's DNS, so every new connection first has to *resolve* memcached2. libmemcached reports the resolver's "temporary failure" as a timeout, but OPT_CONNECT_TIMEOUT never applies to the DNS lookup. On my machine that lookup takes 6 ms and the request is fast; on yours it is evidently around 3 s, and a request pays it twice (the bootstrap container and the main container each open their own connection) — hence ~6 s. You can confirm with `ddev exec getent hosts memcached2` after stopping the container and timing it.
So from these, found two things:
The pool then remembers the dead server for the life of the PHP worker and re-probes it every 30 s (one failed cache operation per probe), so a server that comes back is used again. The README said not to combine removal with persistent connections; that was wrong without OPT_DEAD_TIMEOUT and right with it.
Added changes to README and a new test to try it. Can you test this configuration to see if it resolves your issue?
Comment #16
o'briatI don't know if
getent hostsis a good test, it takes around 8s to give a response on a healthy and up docker/ddev host.I tried with OPT_DEAD_TIMEOUT and pause: same problem.
I tried with OPT_DEAD_TIMEOUT, pause AND using IPs: same problem (and I got the error message that states that it could not connect to up servers 1 & 2).
Maybe I'm missing something or maybe I should not mix persitency with this test?
By the way, the PHP doc said to set
Memcached::OPT_SERVER_TIMEOUT_LIMITwhen usingMemcached::OPT_REMOVE_FAILED_SERVERSComment #17
o'briatI just discovered that the pecl-memcache plugin also implements a hashing method (and this module supports it).
The test is much better, it seems to works flawlessly with one server missing (on 3).
Although, with 1 server on 3, I got slower response times and errors popup "Notice: MemcachePool::get(): Server 10.10.6.2 (tcp 11211, udp 0) failed with: Network timeout (0) in Drupal\memcache\Driver\MemcacheDriver->getMulti() (line 60 of modules/contrib/memcache/src/Driver/MemcacheDriver.php)." x2
Comment #18
japerryThanks for testing both. I updated the test to check against Memcached::OPT_POLL_TIMEOUT which should help timeout those dead servers faster.
Comment #20
japerryMoving back to fixed with the new test in prep for new release.
Comment #22
o'briatSorry, but I tested the 2.x-dev release and it's still not perfect.
I removed 1 memcached (out of 3) and then average request response time drop from 150ms to 1.5s (but better than the previous 6s).
Although:
The status message is now correct and display that one server is unreachable.
Using host-names is now not slower than using IP when a server is missing.
Persistency seems to have no impact on this point.
My settings:
Comment #24
japerryTweaked the requirements hook to make the check interval configurable and to ensure it doesn't check on every pageload of the status page.
Comment #26
japerryMerged the admin requirements piece
Comment #28
o'briatSorry but the response times are still more than 10 time slower with a missing server.
Some response time (the same setup & settings as above):
Also the error message explain that "...The site is running without this cache while the server is unreachable..."
Shouldn't it say instead: "...The site should run fine without this server, but with a smaller cache size, all its cache keys will be rebuild into the remaining ones (if any)."?