Most sources agree that in Solr, using <optimize> commands isn't necessary and just a big waste of ressources.
We therefore should remove our cron job of optimizing (by default) every day. The internal connection class method can stay, but we shouldn't use it in the module. If someone wants to use it, they'll have to add custom code.
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | 2099559-8--optional_optimize.patch | 4.44 KB | drunken monkey |
| #1 | 2099559-1--remove_optimize.patch | 1.49 KB | drunken monkey |
Comments
Comment #1
drunken monkeyPatch attached.
Since we never set it, I don't think we need to remove the variable in an update hook. Or should we still do it?
Comment #2
damienmckennaYes, definitely best to cover your bases and remove it.
Comment #3
drunken monkeyHm, OK.
One thing I didn't think of, though, was that by default we build the spellcheck index on optimize, so the command does have its use. We should therefore find another way to build the spellcheck index regularly, before we remove the
<optimize>calls.Comment #4
nick_vhWe could send a request to solr to build the spellchecker? spellcheck.build=true. That way we stop the optimize command from running and we get more performance benefits as we reduce disk IO. Thoughts?
Comment #5
drunken monkeyYes, that would probably be the only alternative. We could just run it during cron runs, like we do with optimizing now.
Do you know, though, does this only work on the
/selecthandler? It seems like a bit of a waste to execute a search in addition to building the index (though the effect is of course minimal).Comment #6
nick_vhThe tricky thing with manually defining when to build the spellcheck is that we are then *I think* responsible for building all the possible spellcheck libraries and you have no means of knowing which ones those are due to the solrconfig_extra file. I'm not sure what it does if you don't specific the spellcheck.dictionary...
"&spellcheck=true&spellcheck.build=true&spellcheck.dictionary=default."
I we do a select?spellcheck.build=true etc.. request it's not a big deal and you shouldn't care about the results. it won't execute the text search as far as I know.
Comment #7
drunken monkeyOh, OK, that would be good to know, then, of course …
If it really is necessary to specify them, maybe we can just parse the
solrconfig.xmlto see which ones there are? Not terribly ressource-friendly, but if we do it once a day I guess it should be OK.Ah, I'll just have to test this, some time in the next … decade. I.e., when I have time.
Comment #8
drunken monkeyHad a very productive weekend and managed to look into this.
It seems to default to "default" as the dictionary, but still only build the one. Which of course makes sense, as you can also only use one for the actual spellchecking, so providing more than one
spellcheck.dictionaryoptions makes no sense.However, I think we should probably just document that we only build the default dictionary automatically. If someone wants to add their own dictionary, they should take care of building it themselves. (Also, as said in #2145969: solrconfig_extra.xml not working as it should, I don't think this would work currently anyways without modifying the config file directly.)
Or, if this is more common, we can of course also add a
search_api_solr_spellcheck_dictionariesvariable, defaulting toarray('default'), and looping over that. I don't think it's necessary, though.Hm, on the other hand, just allowing users to re-enable optimizing could also solve this easily for them.
It will. But if we set
rows=0and don't include any query or filters, the performance impact will probably be negligible.Anyways, attached is a patch that would fix this by only building the "default" dictionary once a day normally, but allowing the user to change this back to a daily "optimize" via a variable (or to disable this completely, if the dictionary is not used).
(Note to self: document this in the handbook upon committing.)
I also noticed that we currently filter by
class = 'search_api_solr_service'there when loading the servers, so we currently don't execute either operation for subclasses (like Sarnia and Acquia Solr). Do you think we should fix this, maybe by just looking for agetSolrConnection()method?Comment #9
drunken monkeyCommitted. Thanks for your help and input!