errors like Failed to set key: cache_content-content%3A148498%3A148498 [#355299]

im running memcache 5.x-dev on latest d5 stable.

however occasionaly im seeing thousands of entries in watchdog like

 memcache         | Failed to set key: cache_content-content%3A148498%3A148498

phpinfo() tells me im running php 5.2.6 and memcache info

Active persistent connections 	1
Version 	2.2.4
Revision 	$Revision: 1.104 $

i can telnet locally to memcache on port 11211 and store/get keys pairs

i have my settings.php set with


$conf = array(
  'cache_inc' => './sites/all/modules/memcache/memcache.inc',
  'memcache_servers' => array('localhost:11211' => 'default'),
  'memcache_bins' => array('cache' => 'default'),
);

any ideas???

Comments

Comment #1

dgtlmoon commented 13 January 2009 at 01:11

Priority:

Normal

» Critical

I am starting to see this on every server i've setup memcache on, more details to come

Comment #2

pletcher commented 20 February 2009 at 09:36

I'm also getting this error, have you found anything?

If there's any info I could provide, please let me know.

Thanks,
jrp

Comment #3

dgtlmoon commented 15 March 2009 at 14:17

I noticed my Drupal installation has some very long URL's for talking to facebook Oauth etc.

I used tcpdump and wireshark to look at the packets hitting the memcached bin.

memcached bin was returning "CLIENT ERROR bad command line format" - a little bit of googling indicated this could be caused by a bad key string, in this case either it's too long, or there are some dodgey char's in there, i took the too-long option..

Index: memcache.inc
===================================================================
--- memcache.inc        (revision 478)
+++ memcache.inc        (working copy)
@@ -15,6 +15,9 @@
  *   'cache_menu', 'cache_page', or 'cache' for the default cache.
  */
 function cache_get($key, $table = 'cache') {
+  if(strlen($key) > 80) {
+    $key = md5($key);
+  }
   return dmemcache_get($key, $table);
 }

@@ -46,6 +49,10 @@
 function cache_set($cid, $table = 'cache', $data, $expire = CACHE_PERMANENT, $headers = NULL) {
   $time = time();

+  if(strlen($cid) > 80) {
+    $cid = md5($cid);
+  }
+
   // Create new cache object.
   $cache = new stdClass;
   $cache->cid = $cid;

so it seems cache_page is where it lands, and it uses the URL as the key, which in this case is far too long, so i took a punt and just md5'ed the key, the chances of collisions here are extremely low so i'm not fussed

i wonder what the actual key length should be? ive seen even relatively short keys returning errors.

Follow these steps and atleast see if you are hitting your memcached bins
further than that, see what the errors are with

"tcpdump -i lo "port 11211" -s 0 -w out.bin" then use wireshark to search for where you are setting that key you saw the error in your watchdog on, and see what memcached replied

phew :D hope this helps

Comment #4

dgtlmoon commented 15 March 2009 at 23:56

Priority:

Critical

» Normal

Might need some better limiting of key lengths, checksums or something..

Comment #5

pletcher commented 18 March 2009 at 18:41

Nice, thanks. I appreciate you posting your debugging method.

I dug around a little bit, and found this thread: http://lists.danga.com/pipermail/memcached/2007-January/003411.html

The key limit is currently 250 bytes, because key length is getting stored in a single byte inside memcached. Inside that thread they suggest just md5'ing (or md4, or sha1) it, so you get some legitimacy added to your punt.

Is $cid in this case utf-8? If thats the case, the 70-80 character limit makes sense, otherwise it should be 250, correct?

Thanks,
jrp

Comment #6

dgtlmoon commented 26 March 2009 at 04:55

good point re 80char UTF-8, it was just a number i plucked out of the air that 'seemed safe', that said, you could just force an md5sum for all key's, md5 sums are meant to be fairly safe against collisions (assuming your content set isnt too huge, so theres not much risk)