Problem/Motivation
The Tokenizer processor has an input for Ignored characters which may be adjusted to suit a site's particular content. If the input is empty then instead of no characters being ignored, the default ignore characters ._- are removed. Although consistent with the behaviour of the Whitespace characters it is unexpected behaviour based on the help text.
For my particular use case I don't want to use any of the default ignore characters, so that the filenames interesting-document.pdf and interesting_document.pdf are still searchable (ie, they don't both become interestingdocumentpdf). See also related issue 2965995 for another user's similar request.
Steps to reproduce
- Add file to site called
interesting-document.pdf. - Default install of search_api.
- Add index on File datasource.
- Add the Tokenizer Processor.
- Delete all content from the Ignored characters input box.
- Index content
- Search for 'interesting'. The file
interesting-document.pdfis not in the results.
Proposed resolution
Allow users to have no ignore characters by updating the prepare function in web/modules/contrib/search_api/src/Plugin/search_api/processor/Tokenizer.php (the else branch below):
if (!isset($this->ignored)) {
if ($this->configuration['ignored'] !== '') {
$this->ignored = str_replace('/', '\/', $this->configuration['ignored']);
}
else {
$this->ignored = '._-';
}
}
| Comment | File | Size | Author |
|---|---|---|---|
| #2 | 3253986-2--tokenizer_ignore_nothing.patch | 3.29 KB | drunken monkey |
| #2 | 3253986-2--tokenizer_ignore_nothing--tests_only.patch | 647 bytes | drunken monkey |
Comments
Comment #2
drunken monkeyYou’re right, this doesn’t make much sense. We already default to
._-for the setting, why also overwrite an empty value with that? Seems we kept too close to the existingspaceskey when adding this, for no good reason.With that being the default value anyways, I’m pretty confident we won’t break anything by fixing this. (To be on the safe side, I’ll also add something to the next release notes.) The description needs to be adapted then as well.
Attached is a patch implementing this and also providing a test. Please test/review and I can commit it.
Comment #4
gaddman commentedThanks @drunken-monkey! I've tested your patch and works a treat:
bob-the-builder.bob- no results.bob- found postbob-the-builder.Comment #6
drunken monkeyThanks for testing, good to hear it worked.
Committed.
Thanks again!