If you have a db backend index that has stemmer enabled and then add a field, the stemmer preprocessor will have that field unchecked. So the words in that field do not get indexed as normalized 'stems'. However, when you do a full text search that includes both stemmed and unstemmed fields, only the stem is searched for.

More specific example is:

Have a blog author content type and a blog post type that has title, body, and blog author files that is an entity ref to blog author type.

Create an blog author with title of "XXX Roythorne" <- specific item that caused problem
Create a couple of posts with Roythorne as the blog author.

Create an DB index with the title, body and "Blog author » Content » Title" as fulltext fields. Enable the stemmer preprocessor, but only for title and body. Save / index.

Create a full text search index view using the Title, Body, and Blog Author fields.

Search for 'roythorne'...

Nothing will be found.

Dumping the queries made, showed that the stemmer was making the keyword stem be roythorn while the DB contained roythorne for the blog_author field.

A work around was to make sure the blog author field was checked and re-index so the stemmed words are stored.

What should the correct behavior be?

Seems like if there are full text fields that are not stemmed, the search code should search for BOTH the full word and stemmed word and not just the stemmed word. e.g. word in ('stem', 'stemmed'). But that may mess up the count.

At minimum, hopefully this issue will point others to the work around (or proper configuration depending on your P.o.V. :) )

CommentFileSizeAuthor
#4 2859683-3--processors_fields_setting_warning.patch908 bytesdrunken monkey

Comments

cgmonroe created an issue. See original summary.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Title: DB Stemmer search assumes all text fields are indexed as stemmed » Processors don't correctly preprocess keywords per field
Component: Database backend » Framework
Issue summary: View changes

Thanks for reporting this problem!
You're right, this currently won't really work correctly, and it's actually two separate problems:

  1. Whether a processor is enabled by default for a field shouldn't depend on whether the processor or the field is added first. (Although this is debatable and might lead to undesired behavior on some sites.)
  2. While index-time preprocessing can be configured per field, search keywords can't be preprocessed per field, which means that if you search across multiple fields with different processor settings, the resulting search won't be entirely correct. (This is actually a problem for all fields-based processors, not just the Stemmer.)

Since I think those two problems should be discussed separately, I've created #2867809: Add new fulltext fields to enabled fields processors for the former, and modified this one to only describe the latter.

As for solving this problem, it's unfortunately a bit complicated. These are possible solutions I could think of:

  1. As you say, detect this case (search across fields with different states for this processor) and do an OR between both versions. This would already be possible with the current framework. Potential problem: with several processors enabled only for some fields, this would get a mess, possibly reduce performance significantly and might also lead to false matches (though this seems an edge case). For example, having "Ignore case" also enabled for only some fields and searching for "Roythorne", we'd get Roythorne OR roythorne OR Roythorn OR roythorn.
  2. Implement keywords processing (and index fields processing) "correctly", on a per-field basis, like proper search engines like Solr do. A search for roythorne in title and author would be parsed into something like title:roythorn OR author:roythorne. Downside: This is not possible with the current API/framework, and it's now too late to really change that without breaking contrib modules, or investing a lot more effort into implementing this in a backwards-compatible way. Plus, it's pretty complicated to implement to begin with, for a problem which I'd describe as "edge case". If you want different processing for different fields, just use Solr. (I know that was not the point of your issue, you didn't want this configuration to begin with – but just discussing this part of the problem, it's a valid response, I'd say.)
  3. As you say, just remove the "Fields" option for processors. This would be a regression (compared to Drupal 7) and a loss in flexibility, so I'd really like to avoid this. (It's hard to know how many people even use this, though. But with around 80k users, currently, there's bound to be a few using any of our features.)
  4. As an alternative, we could just document clearly the potential problems with enabling a processor only for some fulltext fields.

Any opinions on this?
I think I'd actually lean towards just making users aware of the potential problems.

cgmonroe’s picture

cgmonroe commented

I agree on making people aware of the problem. It would have saved me a lot of head scratching.

Maybe the best thing is to do the following:

When a full text field is added, check for processors applied against other full text field.

Enable the matching processors for the new field.

Issue a warning message saying this has been done and to verify the settings.

This would ensure the search settings default to the normal expected behavior for a standard full keyword search across all fields .. while warning people who have special needs that they need to set those up.

Of course the alternative would be to not change the processor settings on the new field but issue a warning message that the settings should be reviewed.

I think most folks would prefer the former because otherwise you have to manually go to each processor and set the fields. IMHO, it would be better for the special case folks to have to do than the average user.

TIA

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
StatusFileSize
new 2859683-3--processors_fields_setting_warning.patch908 bytes

Most of your comment seems to pertain to #2867809: Add new fulltext fields to enabled fields processors, not the new scope of this issue. Please post a comment there.
For making people aware of this problem: does the attached patch look good to you?
We could also add a page to the module documentation (or, maybe better, a section to the "Common pitfalls" page) explaining this in a bit more detail, and then linking to that. Could you maybe add that?

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Component: Framework » Plugins
Status: Active » Needs review
borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented
Status: Needs review » Reviewed & tested by the community

The docs change looks good.

  • drunken monkey committed 33f0392 on 8.x-1.x 
    Issue #2859683 by drunken monkey: Added a note to fields processors...
drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Status: Reviewed & tested by the community » Fixed

OK, then let's just go with that for now.
I've also added a section to the documentation (D7, D8).

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.