I'm working with a site that has a content index with a few thousand nodes in it. While poking around with the index configuration I realized that stemming had not been enabled, so I turned it on and updated the index. The next day I was informed that the search system was no longer returning data as expected, and when I used Solr Devel to examine the query being sent to the server I discovered that the way it processes the keywords has changed. Before the change when I would search for "path/article" it would convert the string into "path article", but after enabling stemming it get turned into "patharticle".

So the question is, why does the stemmer processor do this? The problem appears to stem (cough) from the following line in processor_stemmer.inc:

    $value = implode('', $stemmed);

Shouldn't it instead be the following:

    $value = implode(' ', $stemmed);
CommentFileSizeAuthor
#2 search_api-n2949562-2.patch373 bytesdamienmckenna

Comments

DamienMcKenna created an issue. See original summary.

damienmckenna’s picture

Status: Active » Needs review
StatusFileSize
new373 bytes

This changes the line to use a space separator on the keywords.

damienmckenna’s picture

Title: Why does word stemming change the query so much? » Word stemming breaks the query string
Category: Support request » Bug report

Changing this to a bug report, because it's a bug.

damienmckenna’s picture

drunken monkey’s picture

Component: Miscellaneous » Plugins
Status: Needs review » Fixed

Thanks a lot for catching that, you're of course completely right!
Note, however, that if you're using Solr, you shouldn't use our homebrew Stemmer processor anyways – Solr can do a much better job of that anyways.
Also, when using the Stemmer, it should be active after the Tokenizer processor (but that's even worse with Solr, so that's just a general note), in which case this bug wouldn't be triggered anyways.

damienmckenna’s picture

Excellent, thanks. BTW I opened a follow-on task to improve the UI around this option: #2949899: Indicate that stemmer shouldn't be used with advanced index engines

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.