Spin-off of #946624: Use a cron queue for indexing: The "Index now" functionality on an index's "Status" tab should use a batch instead of just trying to index all (requested) items at once and failing miserably for larger numbers.
| Comment | File | Size | Author |
|---|---|---|---|
| #17 | 1225620--batch_api-17.patch | 18.86 KB | drunken monkey |
| #13 | 1225620--batch_api_for_index_now-13.patch | 18.61 KB | orakili |
| #11 | 1225620--batch_api_for_index_now-11-alternate.patch | 16.55 KB | orakili |
| #10 | 1225620--batch_api_for_index_now-10.patch | 13.31 KB | drunken monkey |
| #8 | 1225620--batch_api_for_index_now-6.patch | 10.75 KB | orakili |
Comments
Comment #1
drewish commentedOn one site I've got it got totally stalled. It's calling search_api_index_items() which returns 0 meaning it's not doing anything but because the remaining stuff keeps showing up it keeps trying. I'm guessing they're all stuck in the queue some place. I guess I need to figure out how to clear the queue so it starts over.Meant to post this over on #1262478: Make "Index now" consume the items in the cron queue first
Comment #2
Yaron Tal commentedI attached a patch (for the 7.x-1.0 release) to temporarily work around the current limitations. Instead of trying to process all nodes and then index them all, the module will now process and index as much nodes as are set to be in a batch (configurable on the settings tab), and send those to the index before starting with the next batch.
In our configuration we have a couple of big nodes, which would make the module crash somewhere in the middle. By patching it like this we can just run it a couple of times with small batches, and when it runs out of memory we at least have all the nodes till that point in the solr index. Real batch support would be better, but I don't have enough time to create that, and this works as a temporary workaround.
I know this won't make it in to core, but it might help some people.
Comment #3
farald commentedSecond that. We need a batch operation here. Indexing 5000 nodes, 200 at a time is time consuming. :)
Comment #4
drunken monkeyIt's a big day for the Search API … Finally, the longest-lasting feature request is set to "needs review"!
Please help me by testing (and review) the attached patch to finally fix this problem!
Comment #6
orakili commentedTested patch #4. The calculation of the limit is done after the batch is set up so it's not used.
Adding a modified version of the patch.
Comment #8
orakili commentedSimilar patch as in #6 + batch indexing via drush.
I've modified a bit the behaviour of the drush function "search-api-index". The limit parameter corresponds now to the number of batch runs and not the number of items to index.
I've tested it to index 400,000+ nodes.
Comment #10
drunken monkeyThanks for the help, orakili!
I have an improved version here (sadly, developed independently of you, as I didn't think to check back here after starting this on Wednesday), which should work completely and also has test cases which are modified accordingly.
One important addition is also that items, which couldn't be indexed, are now freshly marked as “changed”, so we won't try to index them again right in the next batch. (In the worst case, those could accumulate and make any indexing impossible.)
I've also added your Drush modifications. I can't test them here (Drush doesn't work for me), so it would be great if someone else could review them! Also, why do you need your own batch processing function and can't use the "normal" one? I understand the "finished" callback, as it has to use
drush_log()instead ofdrupal_set_message(), but the other one shouldn't be different, right? (We could also consider using a variable to determine what output function to use, to also unify the two "finished" callbacks – but I guess we should better refrain from such magic.)Please also review my newest patch regarding the normal UI!
Comment #11
orakili commentedArf, I've just finished to modify the patch in #8 and adapt the test to handle the use of the batch API.
I used a different callback for drush just because of the "format_plural" function which can not (should not) be used with drush as it calls t().
Just attaching the patch to make sure we don't lose it.
Well I will review yours and combine the best parts and try to unify the callback and finished functions.
Comment #12
orakili commentedAlso maybe it could be interesting to have 2 fields under the "index now" section with the number of batches and the number of items per batch. It would be flexible. Same for the drush function. What do you think?
Comment #13
orakili commentedHi,
Here is the patch merging both #10 and #11 with some modifications.
Mostly it contains:
I've tested both the UI and the drush function with 10,000+ items.
Comment #14
randallknutson commentedPatch #13 worked for me. Thanks for saving me tons of work!
Comment #15
Ace Cooper commentedLove patch #13, thank you orakili!
Comment #16
Anonymous (not verified) commentedI have also tested the patch from #13 with custom entities and a custom datasource.inc
It works fine.
Comment #17
drunken monkeyThe previous version displayed a wrong error message when the entered limit was greater than the remaining items. This is fixed in the attached patch.
I also by default hid the settings of the „Index now“ form, as I guess the defaults are what users will usually want. Does this make sense to you, too?
Thanks for testing, everyone! Please also try out this newest patch, and than we can hopefully finally fix this!
Comment #18
Anonymous (not verified) commentedWorks fine!
Comment #19
orakili commentedPatch in #17 works fine. Flagging as RTBC.
Comment #20
jsacksick commentedIs there any particular reason for this to stay in RTBC ? :p
Comment #21
drunken monkeyCommitted. Thanks for reviewing!
Comment #22
joelrosen commentedAny idea when we might expect to see this feature brought into the stable branch? I have a client who's asking for it but I'm hesitant to switch over to the dev branch. Thanks!