Closed (outdated)
Project:
Search API
Version:
7.x-1.x-dev
Component:
Framework
Priority:
Minor
Category:
Feature request
Assigned:
Unassigned
Reporter:
Created:
26 Jul 2011 at 10:28 UTC
Updated:
10 Jan 2019 at 08:42 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #1
drunken monkeySounds sensible. Any other opinions (or "+1"s) on this?
(And hm, does this also need the "D7 stable release blocker" tag, as it is an API change? Probably … :-/)
Comment #2
Shadlington commentedThis would be useful.
At some point in the future I will be adding fivestar voting to some of my content which would have similar requirements.
Comment #3
klausiDoes this really qualify as stable release blocker? Why is it a problem if the item is indexed again immediately on the next cron run? I'm bold and just remove the tag, feel free to add it again if you have a plan and want to work on this soon.
Comment #4
drunken monkeyIf we want to implement it, then yes, because it would need substantial API changes to implement.
Probably because of performance? When there are lots of votes, this would index a large number of items at each cron run – apparently unnecessarily for this use case.
However, I'm not really sure we should implement this. It would be a (probably disproportionally large) API change, and we are slowly going into a bit of a feature creep. This would probably not be a very common use case, and with a bit of code (a custom data source controller) it should already be possible.
Moreover, you can just change the cron batch size and the time spent in the cron queue to control how many items are indexed. Then, when there are lots of votes, the system will automatically take some time to index all of them. And when there are few votes, there wouldn't really be a need to wait for indexing, and with the current system it wouldn't.
So, unless there are good arguments in favor of making this an explicit feature, I'll mark this "won't fix".
Comment #5
ralf.strobel commentedWell, it would not only index items unncessesary often, but it might also neglect new items that really should be indexed more urgently.
You should combine this with a function that guarantees some fair attention to all indexed items... On a busy site, it might just happen that all items get updated all the time and the system won't even be able to reindex them that quickly. Right now the API just tries to reindex them all, starting at the lowest item number. In a worst case scenario, it would never index the last ones in the list, because it can only cover 50 items or so per cron run, and by the next time cron runs, the first items might already have been changed again.
This is actually a real bug, not just a feature request.
-----------------
I suggest adding a timestamp column "last_indexed" to the table "search_api_item" and append it to the index "indexing".
When you build the query for the next few items to be indexed, add a condition
[last_indexed < REQUEST_TIME - min_reindex]
and
[ORDER BY last_indexed ASC].
I don't think those changes are that substantial, but they sure are important.
Comment #6
drunken monkeyYou are right, this seems to be a bug introduced in #1064884: Add support for indexing non-entities. I'll fix that immediately.
Comment #7
drunken monkeyComment #8
drunken monkeyYour conclusion regarding the worst case scenario is still wrong, however, as all unindexed items will be inserted into the cron queue at once, and then processed on a FIFO basis. If items with lower IDs are later changed again, this won't prevent the items already in the queue from being indexed before them.
Comment #9
ralf.strobel commentedOh, you're right. I didn't see the "changed" column was already a timestamp. I thought it was just a boolean. Sorry!
(For my suggestion to work it would have to be a boolean, actually.)
--------
Regarding your patch: I don't think that was a very good solution. The new query now uses a sorting for which there is no suitable table index, since "indexing" only consists of "index_id" and "changed".
To be honest I don't understand why it doesn't index new items before changed ones in the current version, since they get inserted with "changed = 1", which is lower than any timestamp.
Comment #10
ralf.strobel commentedAlso, I don't agree with you changing the title of this issue. My feature request still stands. Although you have made a point that this is not as crucial as I had thought. So feel free to defer it until after the release of 1.0.
My initial point was that I would want to save the indexer some unnecessary work for microchanges.
Comment #11
nick_vhIn a sense of bringing old issues back to the surface, has there been any change in the recent search api codebase that fixes this?
I remember drunkenmonkey telling me that search api does not index the same entity in the same index cycle multiple times. In all other cases this seems to be related to #1229790: Allow Search API to index partial entity updates such as statistics for supported backends because what you really want is partial updates.
Adding a time to wait before re-indexing goes against most systems and can cause weird user interaction bugs. Site administrators won't be able to understand why a certain entity is correctly indexed and why the other one is not?
It seems that this issue is only created due to the fact that certain entities get massive amount of updates and others do not. You could however set a cron job and only index entity_type X every 10 minutes and another cron job to only index entity_type Y now.
if not, please correct me. It sounds like a configurable index flow is more important than marking certain entities to "wait". If ralph agrees with this and drunkenmonkey confirms this is already possible we should close this feature request.
Comment #12
legolasboThis issue has not seen activity in over 2,5 years. I am therefore closing this issue to clean up the issue queue. Feel free to re-open and update this issue if you feel this issue is still relevant and of importance.