SearchIndex::updateWordWeights() contains this line:
$or = $this->replica->condition('OR');
foreach ($result as $word) {
$or->condition('word', $word->realword);
}
This query can exceed SQL limits for the length of SELECT clauses, and in some cases PHP memory limits.
Convert to an IN() query.
Potentially chunk the words into multiple smaller queries.
Or use a delete with a subquery instead of building the array in PHP.
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | 1365786-replace-OR-with-IN-clause-for-search-index-update.patch | 1.18 KB | klokie |
| #6 | 1365786-replace-OR-with-IN-clause-for-search-index-update.patch | 1.15 KB | klokie |
Comments
Comment #0.0
ifish commented.
Comment #1
brandy.brown commentedI don't know if this will solve your problem (or if you already did this), but you might need to up your php limit in settings.php and php.ini -- this solution can be found here: http://drupal.org/node/76156
Comment #2
brandy.brown commentedComment #3
brandy.brown commentedComment #4
dddave commentedComment #6
klokie commentedHi, hope you don't mind me reopening this issue but I think that it should be possible to update the search index via cron in under 256 MB of RAM ;)
I've attached a patch that does that, by creating an "IN" rather than an "OR" clause.
Comment #8
klokie commentedoops, resubmitting patch
Comment #9
klokie commentedComment #9.0
klokie commented.
Comment #10
jhodgdonThe patch here (with a quick reroll) would seem to be relevant to D8 as well -- the code is exactly the same in 7 as it is in 8.
The problem is that the function search_update_totals(), which runs at the end of the cron run, is trying to find all the words that need to be removed from the search index, and it's doing it via a db_or() instead of an IN. Probably what it really needs to do is make sure not to do more than about 100 at a time, because you can easily run out of placeholders in such a query. Either that or do a db delete with a join, which is possible to do.
Here's the code that has the problme:
Comment #11
nick_vhSo you are saying to keep a count of the words to delete and execute that per cron run instead of updating everything in 1 execution? I see that you could also keep a count of the words and send multiple queries to not exaust the IN clause but I'm not sure if that would help Drupal from getting out of memory or timing out?
Comment #12
jhodgdonI did something similar to this in the API module -- well, similar in character anyway, if not the details.
What I did was basically decide that I could only execute about 100 items at a time in a query, due to placeholder limitations. So I did something like this (this is not meant to be final code, but something like this):
Comment #13
jhodgdonI just filed a meta-issue #2367253: [META] Several problems in search_update_totals() to clarify the similarities and differences between this issue and the others around search_update_totals(). I'm hoping we can fix them all together?
Comment #22
catchComment #23
catchComment #29
quietone commentedThe Search Module was approved for removal in #3476883: [Policy, no patch] Move Search module to contrib .
This is Postponed. The status is set according to two policies. The Remove a core extension and move it to a contributed project and the Extensions approved for removal policies.
The deprecation work is in #3565780: [meta] Tasks to deprecate the Search module and the removal work in #3565783: [meta] Tasks to remove the Search module.
Search will be moved to a contributed project before Drupal 12.0.0 is released.