Very Slow Solr server response for ONLY first request [#371858]

I've been experiencing very slow Solr server responses of 12 to 20 seconds for only my first page request after the site has site idle for a few minutes. All subsequent requests are very fast. I've placed some timers in the code and I've tracked it down to the following code in the core drupal_http_request function.

  // Fetch response.
  $response = '';
  while (!feof($fp) && $chunk = fread($fp, 1024)) {
    $response .= $chunk;
  }

So the socket is already open and it's trying to read. As I said this is ONLY happening after the site sites idle for a bit and then all subsequent requests are much much faster. I'm suspecting that this might be an Apache Solr setup issue but I was wondering if any of our Drupal users have experienced this? I'm just using the packaged jetty server and I followed the install instruction given here in this module so I've done no other customizations of the Solr side.

Comment	File	Size	Author
#6	repopulate-371858-6.patch	726 bytes	pwolanin

Comments

Comment #1

JacobSingh commented 10 February 2009 at 02:30

Status:

Active

» Fixed

This is likely solr's autowarming cache. Take a look on the Solr site for it, you'll find out how to pre-fill it and deal with this issue. Hopefully not a huge deal if you have enough traffic to keep your searcher going. Basically, every cron run if you send up docs, it will re-gen the cache on the next search, or by and a query you specify in your solrconfig.xml.

Take a look at the Solr docs, it should be explained (a lot) better there.
Best,
Jacob

Comment #2

dreed47 commented 10 February 2009 at 03:24

Thanks, I'll take a look at that. I see in the cron hook for the apachesolr module it's doing a $solr->clearCache and a $solr->Optimize. Could that also be causing a problem if I'm running very frequent cron runs (like every 15 minutes)

Comment #3

dreed47 commented 10 February 2009 at 04:13

Commenting out the $solr->clearCache line in the cron hook seems to have helped this issue greatly for me. I'm not sure if a clearCache is needed before we run an Optimize and I'm not sure of what affect this will have if I DON'T clear the cache on every cron run. Although, conceptually, it seems that we would NOT want to clear the cache that often anyways.

Comment #4

pwolanin commented 10 February 2009 at 13:36

The cache here only has the Luke data, so it should be pretty low cost to regenerate

Comment #5

dreed47 commented 10 February 2009 at 14:51

Category:	support	» feature
Status:	Fixed	» Active

It definitely has huge performance impacts for my use case. Would it make sense to provide an option on one of the settings page to not clear the cache on every cron run? I've changed this to a feature request.