My setup

I've moved off memcache and onto redis as a cache backend. I have a large site with ~60k nodes, and from a simple test pre-caching all my nodes pages in redis, this gives me roughly ~1 - 1.5GB of memory used. I get a couple hundred new nodes every day, so this will continue to increase as time goes on. I cleared my cache yesterday, and today I'm up to ~600 MB of cache used. My server only has 8GB, so I'd like to be able to keep this at a sane level.

Memcache and pruning

Memcache was good, because I could give it a certain amount of memory, and if the limit was reached it would prune entries. I could configure then my other services to use the remaining ram on my server.

Redis-Server and MaxMemory configuration option

It seems that when using redis-server and this module, most keys are set as permanent. From my reading on the redis-server configuration options, if I were to set maxmemory and redis-server ran out of memory, it would attempt to prune keys with an expiry first, but if only permanent entries were left, it would hard fault.

The Drupal redis module

I noticed that in this module, I believe (correct me if I'm wrong) you can set keys to expire via the page / block cache option, but that only has a max setting of 1 day.

Potential maxmemory settings working with the Drupal Module

I'm interested in setting all my redis keys to an expiry of something like 1 year, as 1 day is too short. Then if max memory is reached, redis will auto-prune entries and I'll be at a stable amount of memory. Ideally the module would uptime expire time when keys are get(), so commonly used keys don't get pruned first.

Would something like this be possible? Or am I completely off point? Or is this already possible, and I'm an idiot?

Comments

j0rd’s picture

Issue summary: View changes
pounard’s picture

For what it worth Redis is a persistant database: this means that it won't give you features such as LRU expiration of entries. Memcache is way better at this level. Another problem is that when Redis hits the memory limit it stops writing and that is all. It only allows read until YOU empty it or add more RAM.

I'd be happy to add such memory checks, but that's not this module responsability to do so.

Now considering that fact about Redis (correct me please if I'm wrong in anything, Redis evolves very fast) I did discuss this problem internally with a few coworkers, and it happens that the niciest solution would probably be to set a lifetime for *all values* without exception, which could vary from a few weeks to years depending on the use case, environment, site admin.

So we all agree your solution is the right one, let's implement it :)

This will need to be implemented globally with possibility to set expiry time per bin.

Please note that implementing this in the backend directly is only a safety workaround: in real life you can configure the "cache_lifetime" Drupal core variable to any possible value and all your cache entries will be temporary de facto.

But because I like challenge and because most people won't configure something that isn't really well documented I will implement this failsafe.

If you need it to be more tricky than what I exposed upper (i.e. set a default very long lifetime, configurable per bin) please speak now.

pounard’s picture

Assigned: Unassigned » pounard
Category: Support request » Task
Priority: Normal » Major
omega8cc’s picture

AFAIK Redis does have LRU built-in support - at least in current stable:

# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
# is reached. You can select among five behaviors:
#
# volatile-lru -> remove the key with an expire set using an LRU algorithm
# allkeys-lru -> remove any key accordingly to the LRU algorithm
# volatile-random -> remove a random key with an expire set
# allkeys-random -> remove a random key, any key
# volatile-ttl -> remove the key with the nearest expire time (minor TTL)
# noeviction -> don't expire at all, just return an error on write operations
#
# Note: with any of the above policies, Redis will return an error on write
#       operations, when there are not suitable keys for eviction.
#
#       At the date of writing this commands are: set setnx setex append
#       incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd
#       sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby
#       zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby
#       getset mset msetnx exec sort
#
# The default is:
#
# maxmemory-policy volatile-lru
# LRU and minimal TTL algorithms are not precise algorithms but approximated
# algorithms (in order to save memory), so you can select as well the sample
# size to check. For instance for default Redis will check three keys and
# pick the one that was used less recently, you can change the sample size
# using the following configuration directive.
#
# maxmemory-samples 3
pounard’s picture

Oh nice, didn't knew about that.

Still I'd be confortable with implementing the all-keys-must-expire-one-day feature in the module, it's still be a failsafe for people not using their Redis instance only for cache.

j0rd’s picture

Just to re-iterate so we're all on the same page, so long as some kind of expiry is set on all keys, redis can prune entries when maxmemory is reached as per configuration in MAXMEMORY POLICY posted by omega8cc.

--

As you mentioned Drupal does have a setting for cache_set() with regards to expiry time. Problem is by default it's CACHE_PERMANENT and using permanent (persistent as per redis talk) keys in redis, doesn't allow us to prune entries via maxmemory.

As a solution I would recommend simply setting CACHE_PERMANENT with regards to redis to have a configurable long expiry (default 1 year or something), which would allow redis to prune entries via maxmemory policy if that's what's wanted by the sysadmin.

Now my problem is, I do have 1 bin in my configuration, which I never want to prune, so I still need a capability of putting in keys with out an expiry. Although, I could/should probably persist these to database. Expiry time of 10 years would suffice for that bin.

So I think the solution, as you mentioned, is allow configurable expiry on each bin, which gets used for "CACHE_PERMANENT".

I think that's probably good enough for most everyone, unless someone else has suggestions.

pounard’s picture

I think that what omega8cc pointed out is that you can also configure Redis to drop permanent keys; This is probably not something wise to do if you use Redis for something else than caching.

I think that in #6 you are rephrasing some details that I gave. What I propose is to have one variable per bin (exactly the same as the flush mode) for default TTL for permanent items, which then will become non perment; If the variable is not set, the module will use something like 3 months or so (globally configurable of course) and which could be set to "0" (make them permanent back again).

For example, we could imagine this configuration:

// Make CACHE_PERMANENT items being really permanent
$conf['redis_ttl] = 0;
// OR
// Make CACHE_PERMANENT items live only three months
// (parsing is easy to implement using \DateTime) 
$conf['redis_ttl] = "3 months";

// Make CACHE_PERMANENT items in 'cache_page' bin live for one year
$conf['redis_ttl_cache_page'] = "1 year"

// All non configured bin would inherit from the global
// 'redis_ttl' value

What do you think of that? Would it solve your problem?

pounard’s picture

Ok got this right, I think.

See this commit http://drupalcode.org/project/redis.git/commit/f9f0bf9 (unit tests are don't fully cover but I think it's enough).

Please read documentation about the new feature there: http://drupalcode.org/project/redis.git/blob/refs/heads/7.x-2.x:/README....

Releasing 7.x-2.6 within the day.

pounard’s picture

Status: Active » Fixed
pounard’s picture

Release 7.x-2.6 is 'en route' and coming soon containing this bugfix.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

  • Commit f9f0bf9 on 7.x-2.x, 7.x-2.x-path by Pierre.R:
    Fixes #2144317 - asked by j0rd, helped by omega8cc - Add default TTL for...