Option to compress data

Seems a nice place to explore. How about the performance impact and how does it compare with igbinary ?

I want to see more numbers before adding this, although it remains an optional implementation, why not. Still, it makes it harder to debug by querying Redis directly.

See me some more figures and I might merge it! Thanks for the effort, I'll think about this on my side and talk to my colleagues while you gather some numbers :)

Related core issue: #1281408: Add a compressing serializer decorator

We have tested this patch in production on 46 various VPS-es since November 15, and it drastically lowered RAM usage, without any detectable CPU overhead -- see the screenshot attached.

Big +1 for adding this feature.

If more than one person is interested in this patch, I'm not against pushing it into the module, but it will remain an optional behaviour.

I'm do agree with Alex that it could be generic in someway and be applicable to any cache backend, but in the meantime, the patch is simple enough for me to be added to Redis.

I just need to plug unit tests with the compressed version to see how they behave and ensure there is no regression.

@memtkmc Is the reported CPU usage the one of the Redis' server ? If so, PHP CPU usage would be more relevant.

@pounard It is a total CPU usage on production VPS-es with running PHP-FPM, Nginx, MariaDB, Solr/Jetty and Redis. We couldn't notice any difference in the overall CPU usage after enabling compression in Redis.

Ok, thank you both for your answers. I will add this optional feature and run some tests myself. Thank you again!

Okay, stable enough for me. I did extends all unit tests to verify this works gracefully (only extended them with PhpRedis but it's a tranversal feature that should work with it too).

Alex, for the sake of consistency, I just mostly reformatted your code to my own conventions, but that's still your code and you're officially the author of this patch.

I added two variables for sites admins to control the compression behavior:

Additionnaly, you can alter the default size compression threshold, under which entries will not be compressed (size is in bytes, set 0 to always compress):

  $conf['cache_compression_size_threshold'] = 100;

You can also change the compression level, which an positive integer between 1 and 9, 1 being the lowest but fastest compression ratio, 9 being the most aggressive compression but is a lot slower. From testing, setting it to the lower level (1) gives 80% memory usage decrease, which is more than enough.

  $conf['cache_compression_ratio'] = 5;

Thank you very much to all, this is indeed a very nice patch and idea, and we'll start using it ourselves on our not-yet-in-production projects.

Thanks again!

Release 7.x-3.13 is on the move, and will appear soon. You both have been credited on the project home page, thanks.

Thank you, but it is mostly your work!

Why was it removed in D8 version?

Here's a port for anyone looking for this functionality in Drupal 8. To enable compression, add following configuration to your environment's settings.local.php.

$settings['redis.compression']['enabled'] = TRUE;
$settings['redis.compression']['size_threshold'] = 100;
$settings['redis.compression']['ratio'] = 1;

If the maintainers find this useful, they can reopen this issue.

Please open a new issue, also the 8.x version already integrates with the igbinary project, see #2143149: PHPRedis and igbinary support for read performance and memory reductions..

@bceyssens Thank you so much for your code. My test procedures were:

1. Setup redis on my site without compression enabled
2. Log in to redis cli
3. flushall
4. info memory
5. load a page (make note of the useage)
6. flush all
7. Setup redis on my site with compression enabled
8. load the same page
9. info memory

And results showed:

Before your patch

# Memory
used_memory:27129400
used_memory_human:25.87M
used_memory_rss:31567872
used_memory_peak:45463864
used_memory_peak_human:43.36M
used_memory_lua:36864
mem_fragmentation_ratio:1.16
mem_allocator:jemalloc-3.6.0

After your patch

# Memory
used_memory:8508080
used_memory_human:8.11M
used_memory_rss:11976704
used_memory_peak:45463864
used_memory_peak_human:43.36M
used_memory_lua:36864
mem_fragmentation_ratio:1.41
mem_allocator:jemalloc-3.6.0

That is a savings of 17.76M (or 68% memory usage) for just a single test page.

So to dive more into the theory behind this, while igbinary handles serialization of data it does not handle compression of data which is what the patch in #21 provides. This looks like a great commit for the D8 version, +1 on RTBC. I can't reopen this but have opened a new issue here: https://www.drupal.org/project/redis/issues/3188791

Ok, while the patch here works it looks like this is the issue that ultimately added this feature to D8: https://www.drupal.org/project/redis/issues/2991121

Dropping it here for anyone who needs it.