Strange performance with MemcachePool::get

Hello,

I am working on optimizing production setup and while profiling pages with xhprof I found some strange behavior with MemcachePool::get. I am using memcahe 3.0.5 php client and lastest stable memcache drupal module.

On Prod environment I get:

Function Name Calls Calls% Incl. Wall Time
(microsec) IWall% Excl. Wall Time
(microsec) EWall% Incl. CPU
(microsecs) ICpu% Excl. CPU
(microsec) ECPU% Incl.
MemUse
(bytes) IMemUse% Excl.
MemUse
(bytes) EMemUse% Incl.
PeakMemUse
(bytes) IPeakMemUse% Excl.
PeakMemUse
(bytes) EPeakMemUse%
MemcachePool::get 113 0.1% 885,407 40.1% 876,361 39.7% 855,871 42.5% 846,872 42.0% 221,304,824 473.2% 220,091,432 470.6% 53,511,296 91.1% 53,511,296 91.1%

which is almost a 1s for 113 memcache gets, on my staging environment it is 10x faster. I already tried with changing client, daemon and local memcache to dedicated box and the results on prod stays very stable but slow. PHP version is 5.3.3.

There are two strange things in compare to my staging environment:

1) Memory usage is bigger ( 4x ) on Prod
2) I get following notices on Prod Notice: Trying to get property of non-object i MemCacheDrupal->valid() (line 65 )

I have following stats for memcache:

memcached-tool 127.0.0.1:11211 stats
#127.0.0.1:11211 Field Value
accepting_conns 1
auth_cmds 0
auth_errors 0
bytes 306739254
bytes_read 2395513904
bytes_written 939448303
cas_badval 0
cas_hits 0
cas_misses 0
cmd_flush 0
cmd_get 69221
cmd_set 531150
conn_yields 0
connection_structures 10
curr_connections 3
curr_items 521334
decr_hits 0
decr_misses 0
delete_hits 936
delete_misses 6939
evictions 0
get_hits 60676
get_misses 8545
incr_hits 0
incr_misses 0
limit_maxbytes 3813670912
listen_disabled_num 0
pid 1294
pointer_size 64
rusage_system 161.429458
rusage_user 174.960401
threads 2
time 1374225290
total_connections 72924
total_items 531150
uptime 4388058
version 1.4.4

memcached-tool 127.0.0.1:11211 display

# Item_Size Max_age Pages Count Full? Evicted Evict_Time OOM
3 152B 4149629s 1 44 no 0 0 0
6 304B 4149629s 1 94 no 0 0 0
7 384B 4149623s 1 378 no 0 0 0
8 480B 4148222s 1 281 no 0 0 0
9 600B 4148222s 298 519305 no 0 0 0
10 752B 4149629s 1 120 no 0 0 0
11 944B 4149629s 1 63 no 0 0 0
12 1.2K 4149699s 1 162 no 0 0 0
13 1.4K 4149761s 1 123 no 0 0 0
14 1.8K 4149629s 1 95 no 0 0 0
15 2.3K 4149699s 1 46 no 0 0 0
16 2.8K 4145909s 1 165 no 0 0 0
17 3.5K 4149700s 1 83 no 0 0 0
18 4.4K 4149699s 1 68 no 0 0 0
19 5.5K 4149699s 1 47 no 0 0 0
20 6.9K 4149699s 1 38 no 0 0 0
21 8.7K 4149700s 1 21 no 0 0 0
22 10.8K 4149699s 1 27 no 0 0 0
23 13.6K 4149699s 1 34 no 0 0 0
24 16.9K 4149699s 2 38 no 0 0 0
25 21.2K 4149699s 1 12 no 0 0 0
26 26.5K 4149699s 1 17 no 0 0 0
27 33.1K 4149699s 1 17 no 0 0 0
28 41.4K 4149699s 1 7 no 0 0 0
29 51.7K 4149767s 1 3 no 0 0 0
30 64.7K 4149793s 1 11 no 0 0 0
31 80.9K 4148222s 2 12 no 0 0 0
32 101.1K 4234470s 1 2 no 0 0 0
33 126.3K 4145909s 2 7 no 0 0 0
34 157.9K 4149820s 1 3 no 0 0 0
35 197.4K 4149790s 2 2 no 0 0 0
37 308.5K 4149762s 4 7 no 0 0 0
39 482.0K 0s 2 0 no 0 0 0
40 602.5K 0s 4 0 yes 0 0 0

Any ideas what might be wrong?

Thanks