Problem/Motivation

There is no documented way to use persistent connections with this module and MemcacheD extension.
On further diving into the code I found this issue mentioned in and talks about the legacy Memcache extension #822316: receiving a PHP notice "Connection reset by peer .... dmemcache.inc on line 104".

On further reverse engineering, you reach to the actual connection factory method / class that is disregarding the persistent connection settings all together, so there are NO persistent connections on memcacheD extension. This is not documented anywhere.

Note that the legacy extension had persistent connections. On top of that it is already deprecated and not supported on PHP 7.0+. With new Drupal 8.8 dropping support of PHP 5.6 this should be a close to critical issue for people to be able to migrate to the memcacheD and still have persistent connections.

Official PHP docs:

/* Create a regular instance */
$m = new Memcached();
echo get_class($m);

/* Create a persistent instance */
$m2 = new Memcached('story_pool');
$m3 = new Memcached('story_pool');

/* now $m2 and $m3 share the same connection */

In this module's case the implementation follows the non-persistent connections example.
I know it is durable and overall easier to maintain, but in case of:

  1. High latency networks (imagine ANY cloud provider), establishing a new connection could be too costly to have that overhead on every request.
  2. Huge cluster sizes (server counts) will multiply the TCP overhead on each HTTP request.
  3. Authenticated connections.

Proposed resolution

Implement the persistent connections in the module.

Have a way to respect the configuration that is already present from memcache era:

# We generate the ID on server side.
$settings['memcache']['persistent'] = TRUE;

# Or have something like, so the setting is exposed to the users of the system.
$settings['memcache']['persistent_id'] = 'drupal';

Remaining tasks

Discussion, Patch, Benchmarks, etc.

User interface changes

None expected.

API changes

None expected.
This is a pure feature addition and changes in implementation details only.

Data model changes

New setting to manage the enable / disable and configuration of the persistent connections.

Release notes snippet

Added persistent connections for the MemcacheD PHP extension.

Issue fork memcache-3058121

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

ndobromirov created an issue. See original summary.

ndobromirov’s picture

Here is a nice tutorial to have persistent connections in.

I have PoC for this, but my problem is related to the fact that I can not grasp the idea of the flush method that is part of the interface
MemcacheDriverFactory::flush()

If this is executed on every request, then it's meaningless to have persistent connections implemented at all, unless the whole module is changed to not do this.

ndobromirov’s picture

Status: Active » Needs review
StatusFileSize
new3.38 KB

Here is a PoC for this + docks update on how to configure it.

ndobromirov’s picture

ndobromirov’s picture

Issue summary: View changes
dkarso’s picture

StatusFileSize
new7.51 KB
new3.38 KB

Thanks for the patch ndobromirov, it works good on our setup and we see a small increase in performance. I added a section for redistributing keys when the users uses Libketama and there is an error when setting a key because one of the memcached server is unavailable.

dkarso’s picture

Regarding the #2 comment about MemcacheDriverFactory::flush() - I found this method is only run when the statistics output is requested by the MemcacheAdminSubscriber - When that stats option is disabled, during regular usage, the flush is never called. We can also see via netstat that the connection count is stabilised.

vurt’s picture

I did a test of #6 and it worked fine. My benchmark showed that the site was around 25% faster (local db and memcached).

ndobromirov’s picture

There seems to be some leftovers from the patch in 6 that has changes in the info.yml file.
I think it needs a re-roll, otherwise - a good iteration!

ndobromirov’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll
emerham’s picture

StatusFileSize
new5.94 KB

I re-rolled the patch to not include the changes in info.yml. No other changes where made

vurt’s picture

Tested #11, applied fine.

rpayanm’s picture

Issue tags: -Needs reroll
ndobromirov’s picture

Status: Needs work » Needs review
scuba_fly’s picture

Status: Needs review » Needs work
StatusFileSize
new1.49 KB

FILE: /memcache/src/Connection/MemcachedConnection.php
----------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
----------------------------------------------------------------------
65 | ERROR | [x] Doc comment long description must end with a full
| | stop
----------------------------------------------------------------------
PHPCBF CAN FIX THE 1 MARKED SNIFF VIOLATIONS AUTOMATICALLY
----------------------------------------------------------------------

FILE: /memcache/src/Driver/MemcachedDriver.php
----------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
----------------------------------------------------------------------
26 | ERROR | If the line declaring an array spans longer than 80
| | characters, each element should be broken into its own
| | line
----------------------------------------------------------------------

Time: 119ms; Memory: 8MB

Some code style issues still, and created an interdiff without the info file update because the #6 would not apply against the latest dev version. ( Only difference remains is a moved method ) So code is good, but we need some minder codestyle fixes.

scuba_fly’s picture

Status: Needs work » Needs review
StatusFileSize
new6.18 KB
new2.06 KB
new1.11 KB

Fixing codestyle errors.

hkirsman’s picture

StatusFileSize
new55.42 KB
new7.51 KB
new1.18 KB

I'm not really sure if this works or not but one way seems to be to call isPersistent() method on the Memcached() object.

I thought why not have it on the /admin/reports/memcache next to connections:
is persistent

Is it ok that the "Connections 6 open of 1 104 total" is still growing? On our dev server 1104 is 16 000.

hkirsman’s picture

hkirsman’s picture

StatusFileSize
new53.04 KB
jan kellermann’s picture

Patch #17 works for me. Fine!

Maybe you can choose $settings['memcache']['key_prefix']; as $settings['memcache']['persistent'] per default? In shared environment it could be great to have a pconnection for each drupal. Maybe this could be a hint in the README.

Thank you for your work.

fgm’s picture

Priority: Critical » Normal

You have to be aware of the fact that the design hypotheses of libmemcached (on which the memcached extension is built) are not entirely compatible with PHP, and as a consequence, the behavior of any code using the memcached extension with persistent connections is often unexpected in the presence of server errors, which can lead to the feeling that bugs exist in memcached servers / libmemcached / the memcached extension / this module, while the underlying issue is in fact a hardly documented design choice regarding expectations of behaviour in the face of errors.

I do not have the link at hand, but IIRC there were already a couple of tickets around these issues. The summary being: avoid using the memcached extension and stick with the memcache extension, which is tailor-made for the PHP design choices and behaves as you would expect in the face of errors.

Also, since the memcache extension has been working quite nicely with PHP 8 since 2020 and version 8 of the extension, and features are almost never critical anyway, downgrading from critical to normal

anantaisaustin’s picture

Version: 8.x-2.x-dev » 8.x-2.7
StatusFileSize
new7.49 KB

Changed ReadMe.txt to ReadMe.md

This needs a bit more tweaking than I thought. Will try to get revision up tomorrow. It appeared to be working on my machine but evidently it is not.

anantaisaustin’s picture

anantaisaustin’s picture

Posting the static patch because using the MR doesn't allow pinning to a specific commit, so anyone can submit pretty much anything and inject it into our codebase IIRC. This also fixed our issue.

scuba_fly’s picture

Status: Needs review » Needs work

The md5 in the constructor gives a deprecation warning as the $base_url can be NULL;

scuba_fly’s picture

Status: Needs work » Needs review
StatusFileSize
new7.92 KB

Changing the patch so and empty base url is set to an empty string in the md5()
Interdiff with patch 18 of comment #25
```
- $this->persistentId = $settings->get('persistent_id') ?? $settings->get('key_prefix') ?? md5($base_url);
+ $this->persistentId = $settings->get('persistent_id') ?? $settings->get('key_prefix') ?? md5($base_url ?? '');
```