Stemmer skipping words

So I've looked at this in more detail. Testing different combinations I come up with two instances where $value argument for Stemmer::process contains more than 'word' without spaces. These are: when the tokenizer is not enabled (first) and a field is being indexed; and when a query is being made and it includes "string in quotes".

Without tokenizer, indexing a fulltext field; and query with quotes. Value of $value is like

$value = '<p>Untouched string with spaces</p>';

With when passed through

$words = preg_split('/[^\p{L}\p{N}]+/u', strip_tags($value), -1, PREG_SPLIT_DELIM_CAPTURE);

Results in an array

$words = array(
  0 => 'Untouched',
  1 => 'string',
  2 => 'with',
  3 => 'spaces',
);

Add the tokenizer and you just have

$value = 'Untouched';
$words = array (
  0 => 'Untouched',
);

If the $value string includes multiple space values like:

$value = " if it's not  properly    space \n removed \twhat then  ";

Then the output $words is

array (
  0 => '',
  1 => 'if',
  2 => 'it',
  3 => 's',
  4 => 'not',
  5 => 'properly',
  6 => 'space',
  7 => 'removed',
  8 => 'what',
  9 => 'then',
  10 => '',
);

In all cases here I'm unclear why

PREG_SPLIT_DELIM_CAPTURE

is used, why do you want to catch the space deliminator anyway? And I'm certainly still confused by the mod is used to skip values.

So with the strings I've so far seen being passed into the Stemmer::process it would seem that this makes more sense.

I think I've worked out where the skipping code comes from, if that is correct, then the regex is wrong.

The code on line 113 is:

    $words = preg_split('/[^\p{L}\p{N}]+/u', strip_tags($value), -1, PREG_SPLIT_DELIM_CAPTURE);

Which returns for the example above:

 $value = 'Untouched string with spaces';

 $return = array (
  0 => 'untouched',
  1 => 'string',
  2 => 'with',
  3 => 'spaces',
)

However if the code was to capture (as I happened to notice the Porterstemmer module for core search does http://cgit.drupalcode.org/porterstemmer/tree/porterstemmer.module?h=8.x...

Effectively this with the regex used this would be:

    $words = preg_split('/([^\p{L}\p{N}]+)+/', strip_tags($value), -1, PREG_SPLIT_DELIM_CAPTURE);

Then the example returns:

$return = array (
  0 => 'untouched',
  1 => ' ',
  2 => 'string',
  3 => ' ',
  4 => 'with',
  5 => ' ',
  6 => 'spaces',
)

If the desire is to maintain whitespace between words, then the regex wants changing. Law of unintended consequences suggests this. It does mean, in the core search example linked above that <tag/> angle brackets are returned (I guess no tags names are stemmed) as well as newlines, quotes, spaces etc.

If maintaining whitespace is pointless, then the patch does it just fine.

[ Loop-back issue on what I assume was the inspiration/original code. I think the requirements for core and search api here are a bit different, but for completeness, and for anyone trying to unravel these issues sometime in the future https://www.drupal.org/node/2830484#comment-11796251 ]

Uh why it do that, it was just a note :)