Apart from the plugin-related changes necessitated by D8 (#2044077: Make processors, datasources, service classes into proper plugins) I'd also like to make some changes in the overall way these plugins work together:
- Unify processors and data alterations
- The difference is a bit hard to understand, sometimes you have tasks which need both types and some of the code is plainly duplicated. I think there would be much merit in only having a single type (called "processor", I'd say – or maybe something different entirely, if someone has an idea) taking care of all such functionality. I also have no idea anymore what lead to data alterations being passed the raw items instead of the extracted fields data – from what I see, this has several severe disadvantages. So, just having the current processor's
preprocessIndexItems()method would be a plus in that aspect, too. (Or not – see #2230915: Add additional preprocessing method at index time?.) - Let processors explitly specify which methods they implement
- I.e., add a method which returns a mask of flags for each point at which a processor could be called: for altering the property information, to preprocess indexed items, to preprocess search queries and to postprocess search results. This could help in making it clearer to admins which effect a processor has, thereby maybe also mitigating possible disadvantages of removing the processor/data alteration distinction.
- Pass to item loading and/or wrapper methods whether we're indexing or displaying results
- We could then make it configurable whether we want certain processors (ex-data alterations) to add their fields at search/display time, too. Might be an option, I don't actually know how much sense it makes.
- #1720348: Add the concept of query extenders
- Since we're already unifying processors for different use cases, we could also introduce an additional mode in which a processor could be called (and configured): i.e, query-specific instead of index-specific. Since most processors that preprocess the query or postprocess results will already be suited for being configured on a query-specific basis, I think unifying those concepts also makes much sense. Maybe we wouldn't even need to add any extra code to processors – just give users a UI to easily change the configuration of all processors that run at search time on a query-specific basis (i.e., also allow query-dependent disabling of processors enabled on an index).
- Let processors determine whether a configuration change leads to re-indexing
- Currently, whenever a user changes the configuration, we flag the entire index for re-indexing. This is of course insane for large indexes, when some changes don't require this. See #1160166: Processor setting changes should not always require reindexing. – we should provide some method on processors to let them determine if a change requires re-indexing. Also, with addition #2 we can check whether the processor is actually called during indexing, also providing some kind of sanity check.
Another suggestion would be #1984792: Optionally invalidate index on UI action – maybe we can/should implement that, too. Would be easily possible after a stable release, though. - Allow processors to change the total number of items
- This is a frequently asked question in the issue queue, and certainly not easy to understand for new users. While definitely confusing, it is (I think) a very difficult problem to solve. But maybe with the chance of easily changing the API accordingly, we can come up with a solution.
(This is considerably mitigated by datasources filtering the items directly, though (according to bundles, languages. Also, see the next suggestion for a possible solution.) - Let processors also filter items at tracking time
- In addition to #2044419: Make datasource controllers more powerful, this could be another very flexible and effective way to restrict the indexed items. That way, even if the datasource controller doesn't have the filter setting you need, you can just use a processor and still keep the total item count correct.
- #2090341: Add a way for marking fields/processors as "locked" or "hidden"
- Would make processors more powerful, too.
- Set the status on the processor object
- In addition to constructing the processor with its configuration, we could also pass its status to it, and maybe also notify it when the status changes. Maybe making it generally aware of index changes? (That means the implementation for #2090341: Add a way for marking fields/processors as "locked" or "hidden" could move to the processor class itself.)
Comments
Comment #1
drunken monkey"Unify processors and data alterations" is almost done, I'm just now converting the plugins themselves.
Oh, and the calls in the index have to be adapted, too.
Comment #2
drunken monkeyAdded "Pass to item loading and/or wrapper methods whether we're indexing or displaying results" as a suggestion, but I'm not sure about that one. Just keeping track of things.
Comment #2.0
drunken monkeyAdded "Pass to item loading and/or wrapper methods whether we're indexing or displaying results"
Comment #3
drunken monkeyComment #4
drunken monkeyNew idea: when tracking an item insert/change, we could pass the item to the processors to determine whether it should be indexed. This would keep the total item count correct(er) even when using processors instead of datasource settings for filtering the items.
Comment #5
drunken monkeyAdded #2090341: Add a way for marking fields/processors as "locked" or "hidden", some clarifications and another idea (make processors aware of status and status changes).
Comment #6
drunken monkeyI created child issues for the remaining proposals, closing this one.