Problem/Motivation
There is no documented way to use persistent connections with this module and MemcacheD extension.
On further diving into the code I found this issue mentioned in and talks about the legacy Memcache extension #822316: receiving a PHP notice "Connection reset by peer .... dmemcache.inc on line 104".
On further reverse engineering, you reach to the actual connection factory method / class that is disregarding the persistent connection settings all together, so there are NO persistent connections on memcacheD extension. This is not documented anywhere.
Note that the legacy extension had persistent connections. On top of that it is already deprecated and not supported on PHP 7.0+. With new Drupal 8.8 dropping support of PHP 5.6 this should be a close to critical issue for people to be able to migrate to the memcacheD and still have persistent connections.
/* Create a regular instance */
$m = new Memcached();
echo get_class($m);
/* Create a persistent instance */
$m2 = new Memcached('story_pool');
$m3 = new Memcached('story_pool');
/* now $m2 and $m3 share the same connection */
In this module's case the implementation follows the non-persistent connections example.
I know it is durable and overall easier to maintain, but in case of:
- High latency networks (imagine ANY cloud provider), establishing a new connection could be too costly to have that overhead on every request.
- Huge cluster sizes (server counts) will multiply the TCP overhead on each HTTP request.
- Authenticated connections.
Proposed resolution
Implement the persistent connections in the module.
Have a way to respect the configuration that is already present from memcache era:
# We generate the ID on server side.
$settings['memcache']['persistent'] = TRUE;
# Or have something like, so the setting is exposed to the users of the system.
$settings['memcache']['persistent_id'] = 'drupal';
Remaining tasks
Discussion, Patch, Benchmarks, etc.
User interface changes
None expected.
API changes
None expected.
This is a pure feature addition and changes in implementation details only.
Data model changes
New setting to manage the enable / disable and configuration of the persistent connections.
Release notes snippet
Added persistent connections for the MemcacheD PHP extension.
| Comment | File | Size | Author |
|---|---|---|---|
| #27 | memcache-persistent-connections-memcached-3058121-27.patch | 7.92 KB | scuba_fly |
| #25 | memcache-persistent-connections-memcached-3058121-18.patch | 7.91 KB | anantaisaustin |
| #19 | 2023-05-19_14-01.png | 53.04 KB | hkirsman |
| #17 | interdiff_16_17.txt | 1.18 KB | hkirsman |
| #17 | 3058121-17.patch | 7.51 KB | hkirsman |
Issue fork memcache-3058121
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
ndobromirov commentedHere is a nice tutorial to have persistent connections in.
I have PoC for this, but my problem is related to the fact that I can not grasp the idea of the flush method that is part of the interface
MemcacheDriverFactory::flush()
If this is executed on every request, then it's meaningless to have persistent connections implemented at all, unless the whole module is changed to not do this.
Comment #3
ndobromirov commentedHere is a PoC for this + docks update on how to configure it.
Comment #4
ndobromirov commentedThere is a duplicate in the issue queue, so there are people looking for this missing feature in the module.
#3088085: MemcachedConnection with persistent connections, solution for the implementation + improvement in performances
Comment #5
ndobromirov commentedComment #6
dkarso commentedThanks for the patch ndobromirov, it works good on our setup and we see a small increase in performance. I added a section for redistributing keys when the users uses Libketama and there is an error when setting a key because one of the memcached server is unavailable.
Comment #7
dkarso commentedRegarding the #2 comment about MemcacheDriverFactory::flush() - I found this method is only run when the statistics output is requested by the MemcacheAdminSubscriber - When that stats option is disabled, during regular usage, the flush is never called. We can also see via netstat that the connection count is stabilised.
Comment #8
vurt commentedI did a test of #6 and it worked fine. My benchmark showed that the site was around 25% faster (local db and memcached).
Comment #9
ndobromirov commentedThere seems to be some leftovers from the patch in 6 that has changes in the info.yml file.
I think it needs a re-roll, otherwise - a good iteration!
Comment #10
ndobromirov commentedComment #11
emerham commentedI re-rolled the patch to not include the changes in info.yml. No other changes where made
Comment #12
vurt commentedTested #11, applied fine.
Comment #13
rpayanmComment #14
ndobromirov commentedComment #15
scuba_flyFILE: /memcache/src/Connection/MemcachedConnection.php
----------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
----------------------------------------------------------------------
65 | ERROR | [x] Doc comment long description must end with a full
| | stop
----------------------------------------------------------------------
PHPCBF CAN FIX THE 1 MARKED SNIFF VIOLATIONS AUTOMATICALLY
----------------------------------------------------------------------
FILE: /memcache/src/Driver/MemcachedDriver.php
----------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
----------------------------------------------------------------------
26 | ERROR | If the line declaring an array spans longer than 80
| | characters, each element should be broken into its own
| | line
----------------------------------------------------------------------
Time: 119ms; Memory: 8MB
Some code style issues still, and created an interdiff without the info file update because the #6 would not apply against the latest dev version. ( Only difference remains is a moved method ) So code is good, but we need some minder codestyle fixes.
Comment #16
scuba_flyFixing codestyle errors.
Comment #17
hkirsman commentedI'm not really sure if this works or not but one way seems to be to call isPersistent() method on the Memcached() object.
I thought why not have it on the /admin/reports/memcache next to connections:

Is it ok that the "Connections 6 open of 1 104 total" is still growing? On our dev server 1104 is 16 000.
Comment #18
hkirsman commentedComment #19
hkirsman commentedComment #20
jan kellermann commentedPatch #17 works for me. Fine!
Maybe you can choose $settings['memcache']['key_prefix']; as $settings['memcache']['persistent'] per default? In shared environment it could be great to have a pconnection for each drupal. Maybe this could be a hint in the README.
Thank you for your work.
Comment #21
fgmYou have to be aware of the fact that the design hypotheses of libmemcached (on which the memcached extension is built) are not entirely compatible with PHP, and as a consequence, the behavior of any code using the memcached extension with persistent connections is often unexpected in the presence of server errors, which can lead to the feeling that bugs exist in memcached servers / libmemcached / the memcached extension / this module, while the underlying issue is in fact a hardly documented design choice regarding expectations of behaviour in the face of errors.
I do not have the link at hand, but IIRC there were already a couple of tickets around these issues. The summary being: avoid using the memcached extension and stick with the memcache extension, which is tailor-made for the PHP design choices and behaves as you would expect in the face of errors.
Also, since the memcache extension has been working quite nicely with PHP 8 since 2020 and version 8 of the extension, and features are almost never critical anyway, downgrading from critical to normal
Comment #22
anantaisaustinChanged ReadMe.txt to ReadMe.mdThis needs a bit more tweaking than I thought. Will try to get revision up tomorrow. It appeared to be working on my machine but evidently it is not.
Comment #23
anantaisaustinComment #25
anantaisaustinPosting the static patch because using the MR doesn't allow pinning to a specific commit, so anyone can submit pretty much anything and inject it into our codebase IIRC. This also fixed our issue.
Comment #26
scuba_flyThe md5 in the constructor gives a deprecation warning as the $base_url can be NULL;
Comment #27
scuba_flyChanging the patch so and empty base url is set to an empty string in the md5()
Interdiff with patch 18 of comment #25
```
- $this->persistentId = $settings->get('persistent_id') ?? $settings->get('key_prefix') ?? md5($base_url);
+ $this->persistentId = $settings->get('persistent_id') ?? $settings->get('key_prefix') ?? md5($base_url ?? '');
```