Unsure if this is a bug report or support request, but I'm seeing strange behaviour in specific scenarios and thought I should document it here.

I'm using Redis module 8.x-1.4, PhpRedis client 5.2.1+4.3.0-1+ubuntu18.04.1+deb.sury.org+1 and Amazon Elasticache for Redis 5.0.6, with maxmemory-policy set to allkeys-lru. This works fine, until the Redis instance actually fills up and reaches maxmemory. At this point, evictions do start happening, but I also see the following exception in the logs:

RedisException: read error on connection to redis:6379 in Redis->exec() (line 56 of modules/contrib/redis/src/Cache/PhpRedis.php)

This doesn't happen until Redis is full. It also doesn't consistently happen for all pages, and when it does then refreshing the page almost succeeds.

I tried increasing the Redis instance size, which just made the problem take longer to occur (as it takes longer to reach maxmemory), and also tried ini_set(‘default_socket_timeout’, -1) as suggested in some StackOverflow posts but this didn't help either - it didn't seem to be a timeout issue anyway. None of the monitoring tools available to me at AWS indicate that that there is any kind of issue; CPU load is low, memory usage is high but bumps up and down as items correctly get evicted.

Finally I tried downgrading the server to Redis 4.0.10 and this seems to have made the issue go away. I had a look at the release notes for Redis 5 but can't see any reason why this might happen, I've also searched issue queues here, for other PHP projects and for PhpRedis and Redis itself but can't find any mention of this particular case.

I do wonder if this is perhaps the same as #3117760: Incompatible with twemproxy since 1.2.0 as it's possible that Amazon Elasticache uses twemproxy behind the scenes, but they don't explicitly document this.

Comments

longwave created an issue. See original summary.

longwave’s picture

Issue summary: View changes
berdir’s picture

Hm, sounds like aws support or so might be able to help with that?

This error happening when storage is full sounds like redis might not be configured correctly to drop stuff then?

longwave’s picture

The account this is happening on isn't paying for an AWS support plan at the moment, so there is no technical support available to look at this from the AWS side.

As far as I can see Redis is configured to drop keys, as allkeys-lru is set and the number of evictions does start increasing as soon as maxmemory is reached. I also don't understand why Redis 4 works (with the exact same configuration) and Redis 5 doesn't. For now I am just sticking with Redis 4 but may revisit this in the future, certainly if I start seeing the same issues on Redis 4.

I guess one thing I should try if I have time is install Redis 5 somewhere myself and see if I can reproduce it, as that would tell me if it's AWS's implementation of Redis or not. I could also try using Predis as a client and see if that has the same problem.

yatendrasingh121’s picture

Hi,

Any update on this i am also facing same issue.

shashank.singh11’s picture

Hi,

We are using Redis 5 in drupal 8. Getting same error "read connection error" frequently to user. Please help if anyone fix same error.

tbadaczewski’s picture

I'm using AWS ElastiCache with the Redis 3.2.4 engine and I'm getting notices as well. In the parameter group for the service the maxmemory-policy is set to "volatile-lru", this is a default for ElastiCache. Also, this is under Drupal 8.9.5 with module version 8.x-1.4.

Here are the notice I'm getting 3 times in a row when I access "/admin/reports/redis":

Notice: Trying to access array offset on value of type bool in Drupal\redis\Controller\ReportController->overview() (line 172 of modules/contrib/redis/src/Controller/ReportController.php).
Drupal\redis\Controller\ReportController->overview()
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 573)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 151)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 68)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 106)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 85)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 52)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 708)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)
agoradesign’s picture

I can confirm #7 - happens since we upgraded to PHP 7.4.x

shin.cho’s picture

I have a look at blow URL,
https://aws.amazon.com/premiumsupport/knowledge-center/oom-command-not-a...

When redis memory is max, redis will try to clean same old keys dependens on same conditions,
※allkeys-lru: make space by evicting less recently used keys first
and with mode [volatile-lru],the conditions is ttl, I think we should set the [redis_perm_ttl] in settings.php?

but, this settings in README.md is TODO status?
⇒Chapter ”Default lifetime for permanent items”

ferbdt’s picture

I have the same issue, I've setup redis to use allkeys-lru but as soon as my redis gets full, I can see random errors of:

RedisException: read error on connection to xxxxxxxxxxxxxxxxx:xxxx in Redis->hGetAll() (line 59 of modules/contrib/redis/src/Cache/PhpRedis.php).

I'm using redis 5.0.6 in AWS ElastiCache.

Any suggestion or clue on how to solve this?

longwave’s picture

@ferbdt I started a new Elasticache instance on Redis 4.0.10 and that has been working fine for me for several months now with allkeys-lru, keys are being evicted as expected and I haven't seen any other issues.

ferbdt’s picture

Thanks a lot @longwave, I'll try with redis 4 then!

olli’s picture

We've also seen this with Elasticache redis 5. HTTP 5xx error page was rendered on article form submit.

Should we catch and log RedisExceptions?

alexgw’s picture

After recreating Redis cluster without TLS - I no longer had any issues.
I had defined TLS as below, but was not able to get Drupal to connect, using version 1.5 - connection worked with TLS as expected with redis-cli

$settings['redis.connection']['scheme'] = 'tls';
gerzenstl’s picture

I was getting the same error "read error on connection to ..."

I'm using:
Drupal 9.2.4
PHP 8.011(with PHPRedis)
Redis 6.0.5

I was able to solve the problem by adding "tls://" as a prefix of the value used on the "host" parameter.

Example:

$settings['redis.connection']['host'] = 'tls://{domain of the AWS Elasticache instance}';
casivaagustin’s picture

For the record, We have encounter with a similiar issue, "read connection error"

After several days of debugging, chaninging timeouts, limits and other parameters, there was no luck.

Continuing with the debugging we found that accidentally we were retrieving too much data from redis because a bad query to the database, actuallty we were dumping all cached nodes from Redis.

Long story short, before doing any change in the infrastructure be totally sure that you are not doing something inficient in your code that is consuming your resources in Redis.

Using redis-cli can help a lot to debug what is happening in redis.

With the command monitor, we were able to spot the huge amount of request that drupal were doing to the redis server.

Also SLOWLOG GET 10 was super helpful to find other issues related with heavy queries.

redis-cli --latency can help you to find is there is some connectivity issue.

redis-cli --bigkeys can help you to find big objects

Hope this tips can help, Regards

smustgrave’s picture

We are using
Drupal 9.3
Php 8.0
Redis 6.2.5 and PHPRedis

It connects fine but we get random errors every 20 minutes or so

RedisException: Connection closed in Redis->hGetAll() (line 59 of /var/www/webroot/modules/contrib/redis/src/Cache/PhpRedis.php).

RedisException: read error on connection to tls://XXXXXXXXXX.com:XXXX in Redis->hGetAll() (line 59 of /var/www/webroot/modules/contrib/redis/src/Cache/PhpRedis.php).

Any suggestions?

casivaagustin’s picture

smustgrave, try enableing the compresion, that will reduce the size of the objects to transfer and ofcourse the time to transfer.

Compression
-------------------------
Compressing the data stored in redis can massively reduce the nedeed storage.

To enable, set the minimal length after which the cached data should be
compressed:

    $settings['redis_compress_length'] = 100;

By default, compression level 1 is used, which provides considerable storage
optimization with minimal CPU overhead, to change:

    $settings['redis_compress_level'] = 6;

if the problem persists, try extendiog the Read Timout of the PHPRedis client, you'll have to patch the module adding into the class redis\Client\PhpRedis, inside of the getClient method, before the return, this line

$client->setOption(Redis::OPT_READ_TIMEOUT, -1);

Sadly there isn´t a configuration of the module to set this value into a settings file. By default the library is setting this time to 3 seconds.