Rework the conversion from "fulltext" to tokenized_text

We should make sure this logic is really robust and in the core of Search API. I believe this logic is broken now and it should have tests.

Tests are of course a good idea, but I'm pretty sure this works fine at least in D7. But of course there might have been some errors introduced during the port.

One proposal is to always convert text to tokenized text before sending it to the processors so that the processors can process the text in a consistent manner.

You mean to just convert each text field value to a tokenized version with just a single token, and then let processors split that further, if they want? Might be an idea, yes. However, one problem would then be that it's not readily apparent whether any tokenizing happened on the content, which is, e.g., important for the database backend (which has to fallback to its own tokenizing if handed untokenized text).

Something to consider in any case would be to make a "Token" class for this, instead of using a magic array. We might even want to make a class for each data type we support, encapsulate conversion logic there (instead of having that in each backend separately) and also use those to implement custom data type support (by letting other modules just add new types with their classes).
This would be a lot more work, of course, but also be a lot cleaner and have additional benefits. (E.g., I have currently no idea how we should support custom data types at all.)

Then we could just let the class for fulltext data have some methods for tokenizing the data.

If we want a new way of handling tokenized text, I'd say we first need to identify what use cases it has to cover and make sure there are no problems with one. I can think of the following:

• Change the text as a whole (e.g., to lowercase). Doesn't care whether it gets handed individual tokens or whole field values.
• Split text into tokens. The source can be either tokens themselves, or untokenized field values. In the former case, each processed token should be replaced by the ones it is split into. (This is currently done by very complicated logic in the processor.)
• As a backend, see whether text has already been tokenized (e.g., DB).
• As a backend, use the tokenized text for indexing (e.g., DB).
• As a backend, ignore the tokenization and index each field value as a single piece of text (e.g., Solr).

Your proposal of converting the field values to single tokens even before handing them to the processors would, as far as I see it, solve most of those. The middle one could be solved by a isTokenized() method (if it's an object – one more magic array key otherwise (or, a little cleaner but still ugly, an "extra data" entry on the search item)), with only Tokenizer setting this. (This would also be an improvement in that values that were only run through the HTML filter won't be recognized as "tokenized".) The last one could be solved by an extra method or function doing this conversion.

I'm not a processor expert but the above plans *sounds* ok.

Tokenization can still be very useful for Solr if you want to extract certain things out of your text and give it extra boost. Such as the HTML filter does.

In theory, yes. However, boosting individual tokens doesn't work with Solr, since Solr only takes a boost for the whole field. So either we'd have to use some magic with additional fields, like the apachesolr module, or the HTML Filter only makes sense to strip HTML from the data for Solr.

This seems a lot cleaner to me and passes at least the unit tests.

The basic principle is that the values of fulltext fields are now always TextvalueInterface objects, with methods for getting the text value, the original text value and a set of tokens (if set – tokens are also classed objects). Text values can also have properties that processors are encouraged to set, if it makes sense, so it's easy to determine whether a certain text value has been, e.g., tokenized yet.

Does that make sense to you? Does it cover all our requirements? (Seems that way to me.) Did I forget anything or do you have other suggestions?

Btw: already using #2648050: [policy, no patch] Stop disallowing camelCase for local variables / parameters here, which should be finalized soon enough. Yay for saner setters!

1. +++ b/src/Plugin/search_api/data_type/value/TextToken.php
   @@ -0,0 +1,76 @@
   +  /**
   +   * Implements the magic __toString() method.
   +   */
   

   Let's replace this comment with: "Returns a proper value when this object is printed", because I think everyone can see that this is an implementation of __toString() ;)

2. +++ b/src/Plugin/search_api/data_type/value/TextValue.php
   @@ -0,0 +1,147 @@
   +  /**
   +   * Implements the magic __toString() method.
   +   */
   

   See above.

As per personal discussion with Joris, I left the doc comments as-is but refactored the splitting into words (though unrelated, in truth) into a method so it's more self-documenting.

Tagging this with the dev days tag.

Good idea, thanks!
Test bot is also happy, so: committed.
Thanks again for your review!