Search indexing can bog down and cause cron to fail when indexing using the default settings, especially on sites that are heavy on HTML code in the front end. For example, on a site with a lot of complex data tables (i.e. tons of markup to display a bunch of numbers), cron was failing due to search timing out. This led to "attempting to re-run cron" errors. HTML filter is the likely cause; we just turned it on for full page override searches recently.

There are a few things that can be adjusted in the Panopoly Search feature settings to improve indexing performance. First, Drupal core search is still indexing content 100 nodes at a time per cron run. This is unnecessary. And second, the default setting for Search API has it doing 50 nodes at a time per cron run. Aside from initial indexing of a new site (which can be done in the UI), 20 per cron run should be sufficient for most purposes.

Patch forthcoming.

CommentFileSizeAuthor
#6 panopoly-search-index-performance-2565175-6.patch2.99 KBcboyden
#6 interdiff-2-6.txt1.38 KBcboyden
#3 interdiff-1-2.txt1.75 KBcboyden
#3 panopoly-search-index-performance-2565175-2.patch2.93 KBcboyden
#2 panopoly-search-index-performance-2565175-1.patch1.89 KBcboyden

Comments

cboyden created an issue. See original summary.

cboyden’s picture

cboyden commented
Assigned: cboyden » Unassigned
Status: Active » Needs review
StatusFileSize
new panopoly-search-index-performance-2565175-1.patch1.89 KB

Patch is attached. It sets the core search_cron_limit variable to 10 (the lowest number available in the UI), and the Search API cron_limit option to 20 - unless the cron_limit option already differs from the previous default of 50.

cboyden’s picture

cboyden commented
StatusFileSize
new panopoly-search-index-performance-2565175-2.patch2.93 KB
new interdiff-1-2.txt1.75 KB

Updated patch adds core search_cron_limit variable to strongarm.

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
commented
Status: Needs review » Needs work

Thanks!

Only one small piece of review:

+++ b/panopoly_search.install
@@ -171,3 +171,23 @@ function panopoly_search_update_7007() {
+  foreach (search_api_index_load_multiple(array('database_node_index', 'node_index')) as $index) {
+    // If the index configuration is stored in the database, also update it
+    // there. Only update it if it matches the previous default.
+    if ($index->status & EXPORT_IN_DATABASE) {
+      $options = $index->options;
+      if ($options['cron_limit'] == "50") {
+        $options['cron_limit'] = "20";
+        $index->update(array('options' => $options));
+      }
+    }
+  }

We should wrap this code in a check for module_exists('search_api') because per #2555385: Check that module is enabled in update hooks this (and the search_api module) could be disabled. If the module was re-enabled later, then Features should revert and make the change for us.

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
commented

I just manually tested the upgrade, and it works for me!

And here is the Travis build for this patch:

https://travis-ci.org/panopoly/panopoly/builds/79479448

Assuming that passes, and the fix from #4 is made, then this should be ready to commit. :-)

cboyden’s picture

cboyden commented
Status: Needs work » Needs review
StatusFileSize
new interdiff-2-6.txt1.38 KB
new panopoly-search-index-performance-2565175-6.patch2.99 KB

Updated patch and interdiff are attached. I thought we might put the variable_set inside the module_exists as well - if Search API is not enabled, the site might be using core search and we shouldn't change its settings.

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
commented
Status: Needs review » Fixed

Sounds good to me! Did some quick testing on the latest patch, and everything looked good. Committed, thanks! :-)

  • dsnopek committed bec996d on 7.x-1.x 
    Update Panopoly Search for Issue #2565175 by cboyden: Performance issues...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.