There are currently several open issues reporting problems in the function search_update_totals() in search.module. This function runs at the end of Search cron, and updates the overall word scores for each word that has been indexed, so that when search queries run, this overall word score can be used to rank results. I am making a meta issue here to clarify what is going on, and possibly we can fix them all at the same time.
The underlying problem is that if you have a site with lots of nodes, depending on your search settings, you can run out of memory or have database exceptions when this code is run. And even if you don't, the calculation is not very optimal.
The current Drupal 8 code for this function is:
function search_update_totals() {
// Update word IDF (Inverse Document Frequency) counts for new/changed words.
foreach (search_dirty() as $word => $dummy) {
// Get total count
$total = db_query("SELECT SUM(score) FROM {search_index} WHERE word = :word", array(':word' => $word), array('target' => 'replica'))->fetchField();
// Apply Zipf's law to equalize the probability distribution.
$total = log10(1 + 1/(max(1, $total)));
db_merge('search_total')
->key('word', $word)
->fields(array('count' => $total))
->execute();
}
// Find words that were deleted from search_index, but are still in
// search_total. We use a LEFT JOIN between the two tables and keep only the
// rows which fail to join.
$result = db_query("SELECT t.word AS realword, i.word FROM {search_total} t LEFT JOIN {search_index} i ON t.word = i.word WHERE i.word IS NULL", array(), array('target' => 'replica'));
$or = db_or();
foreach ($result as $word) {
$or->condition('word', $word->realword);
}
if (count($or) > 0) {
db_delete('search_total')
->condition($or)
->execute();
}
}
Here are the currently-reported problems with this code:
Fix the three bugs, and probably the "could be more optimal" task as well.
All issues still need resolution. Some have patches, but probably the approaches on the 3 bugs should be combined.
None.
None.
Comments
Comment #1
jhodgdonActually, given that there are 3 bugs that can cause fatal errors in some sites and some database servers, I think this one is at least Major.
Comment #2
jhodgdonSo... talked to pwolanin about this at DDD today...
Fundamentally, we need to change how this functionality works. And as a note, this is functionality for normalizing scores for sorting results to prioritize overall less frequent words over overall more frequent words.
a) We need to use something like
http://weblogs.sqlteam.com/peterl/archive/2007/09/20/Finding-records-in-...
to figure out if records from one table do not exist in another, not a left join followed by a db_or() too many conditions delete.
Or maybe a different query like
SELECT t.word AS realword, i.word FROM search_total t LEFT JOIN (select distinct word from search_index) as i ON t.word = i.word WHERE i.word IS NULL;
b) We need to make sure the search_totals normalization table is done per type and maybe per language, not combining every search plugin's index together.
c) The "dirty" process itself is not really right. What it's currently doing during indexing is that when a word is added to the index, it is marked as "dirty", so that all the words in a node (for example) that is being indexed are marked as "dirty". These ones are renormalized. However, if you are reindexing an existing node that was updated, the old words from the previous version are *not* renormalized unless they happen to also exist in the new node (or another one that is indexed in the same cron run). So, that is wrong.
Also when a node gets deleted, the words there are no mechanisms for renormalizing the words that had been in that node (which is only important if the words also exist in another node).
So... we need a better way to track "this normalization score needs to be updated", and we need to track it between cron runs (nodes are deleted from indexing when they're deleted, not just during cron runs).
d) The db_or() part is probably silly. Should be using an IN query instead, should be more efficient. Still would need to be chunked.
e) The loop at the top of the function that does foreach (search_dirty() ... should be using fewer queries with chunking. There's a patch on #257916: Improve performance of search totals calculation that does this.
f) There were questions on #257916: Improve performance of search totals calculation about whether log functions were available in all databases. It looks like the function LN is pretty universal (natural log), except in SQLite (but we can define one in PHP) -- even in databases that we don't officially support (SQLServer, Oracle, etc.) -- so that should be fine -- and we don't really care which log function we use as long as it's a log [as long as we always use the same log function]. Database drivers for other databases should be able to substitute something in to convert from one log to another if necessary.
Comment #3
hanoiiI just added one odd bug, It's not related to the rest, and its quite hard to reproduce/follow. It could even be a php issue.
Comment #10
jhodgdonAs a note, this function is now being moved to a service, but most likely all these problems remain.
Comment #13
catchComment #20
quietone commentedThe Search Module was approved for removal in #3476883: [Policy, no patch] Move Search module to contrib .
This is Postponed. The status is set according to two policies. The Remove a core extension and move it to a contributed project and the Extensions approved for removal policies.
The deprecation work is in #3565780: [meta] Tasks to deprecate the Search module and the removal work in #3565783: [meta] Tasks to remove the Search module.
Search will be moved to a contributed project before Drupal 12.0.0 is released.
Comment #21
quietone commentedThe Search Module has moved from core to contrib,
Comment #22
quietone commented