Problem/Motivation

The Tokenizer processor has an input for Ignored characters which may be adjusted to suit a site's particular content. If the input is empty then instead of no characters being ignored, the default ignore characters ._- are removed. Although consistent with the behaviour of the Whitespace characters it is unexpected behaviour based on the help text.
For my particular use case I don't want to use any of the default ignore characters, so that the filenames interesting-document.pdf and interesting_document.pdf are still searchable (ie, they don't both become interestingdocumentpdf). See also related issue 2965995 for another user's similar request.

Steps to reproduce

  1. Add file to site called interesting-document.pdf.
  2. Default install of search_api.
  3. Add index on File datasource.
  4. Add the Tokenizer Processor.
  5. Delete all content from the Ignored characters input box.
  6. Index content
  7. Search for 'interesting'. The file interesting-document.pdf is not in the results.

Proposed resolution

Allow users to have no ignore characters by updating the prepare function in web/modules/contrib/search_api/src/Plugin/search_api/processor/Tokenizer.php (the else branch below):

    if (!isset($this->ignored)) {
      if ($this->configuration['ignored'] !== '') {
        $this->ignored = str_replace('/', '\/', $this->configuration['ignored']);
      }
      else {
        $this->ignored = '._-';
      }
    }

Comments

gaddman created an issue. See original summary.

drunken monkey’s picture

Version: 8.x-1.21 » 8.x-1.x-dev
Status: Active » Needs review
StatusFileSize
new647 bytes
new3.29 KB

You’re right, this doesn’t make much sense. We already default to ._- for the setting, why also overwrite an empty value with that? Seems we kept too close to the existing spaces key when adding this, for no good reason.

With that being the default value anyways, I’m pretty confident we won’t break anything by fixing this. (To be on the safe side, I’ll also add something to the next release notes.) The description needs to be adapted then as well.

Attached is a patch implementing this and also providing a test. Please test/review and I can commit it.

gaddman’s picture

Status: Needs review » Reviewed & tested by the community

Thanks @drunken-monkey! I've tested your patch and works a treat:

  1. Created a post titled bob-the-builder.
  2. Search for bob - no results.
  3. Apply patch.
  4. Search for bob - found post bob-the-builder.

  • drunken monkey committed 4faaa85 on 8.x-1.x
    Issue #3253986 by drunken monkey, gaddman: Fixed empty "ignore...
drunken monkey’s picture

Status: Reviewed & tested by the community » Fixed

Thanks for testing, good to hear it worked.
Committed.
Thanks again!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.