Performance comparison with Apachesolr module and Search API module (with benchmarks)

Wow, thanks for doing this! And great to see Search API perform so well!
It's good to now have some independent material on performance. This might well help a lot of others who are unsure about Search API performance. Thanks again!

Tagging your two performance discussions, to link them on the project page.

(Hard to find a fitting category here, but the issue is probably "fixed" in any case.)

Can you increase the number of nodes?
It would be a great test for sites that have a large number many records.
The test results for 10.000, 50.000, 100.000 nodes should be relevant.

It might be better for sites with a lot of content to have the possibility to rely solely on Solr.

Retrieving entities with attached fields implies join operations that decrease performance.
They are already in Solr, so retrieving them when querying Solr would eliminate the joins.

>Retrieving entities with attached fields implies join operations that decrease performance.

Field API caches those results.

Sorry to say this, but this is a very naive comparison.

Apache Solr is lauded for its scalability i.e. being able to search and index thousands (and possibly millions) of nodes effectively and efficiently. So jumping to conclusions by benchmarking the performances of two search engines based on just 83 nodes is just down right naive.

As @vladady noted, please test with 10,000 (and more) nodes for a better evaluation of performance.

Subscribing, in case someone decides to do more comparisons. I'm very interested in this subject :)

There are a huge number of web sites with pages in the hundreds or low thousands, making small search results relevant. The larger searches might be relevant in a book shop where you forget the name of the author, the book, and search for something else, perhaps "Hogwarts". If a shop sells 10000 computer parts, the shopper selects SSD first then 250GB then searches less than a hundred candidates. If a shop sells five million types of shoes and I select "mens shoes", there will only be three shoes in the search list. For many of my sites, the best test would be a thousand pages, a term selecting one hundred items and the search looking at the one hundred items.

The other consideration is the extra server for Solr. A typical Solr configuration has one server, or VPS for the Web site and another for Solr. If you combine the resources and use just a database search, the database search will run faster.

A valid comparison would be the Web site in a one GB, one GHz VPS with Solr in an identical VPS then compare it to a VPS costing the same as the two separate VPSes. Combining the money will often get you double the CPU power and quadruple the memory. Your whole Web site then screams along and your visitors save far more time than they might save from a small difference between the two searches.

First off, let me mention I'm a fan of Search API, but in my latest project I've been using the ApacheSolr module. So I have experience with both. Much like Search API, I'm actually entity_load'ing my entities after getting the results from solr, just for ease of use and since performance is not yet an issue.

With that said, I've done a lot of performance debugging on my site and I'm going to get my broom and call shenanigans on these benchmark numbers.

--

First off, performing this test on 20 or 10,000 nodes does not matter. 20 is a good result set, as normally you'll be paging the results back from solr at around 20. Both ApacheSolr & Search API perform queries to SOLR and a 100 or 10,000 would really only be testing how each module structures the data in the solr index, not actual performance of displaying a page with the results. I suppose additionally Search API could have the advantage here, as I believe it's possible to limit the amount of fields returned from a document, where apachesolr returns them all. Either way, this shouldn't be what these benchmarks are about. Assuming both modules don't do something stupid by indexing the documents poorly in Solr.

--

Here are my gripes with these numbers. There's lots of factors which seem not to be elaborated on, and thus I don't consider them valid until I hear more details. Thanks again for doing them, but I'm a skeptic. Here's why:

1. Do you have page & block cache off? Results with these caches on, are invalid as they do not test the entire cost of a full un-cached page hit.

2. What time are you measuring? Due to the small 20 ms numbers, I assume you're only measure the time it takes to query solr and not actually the time it takes to render the entire page. Drupal (with out page cache) in my experience can not render a page with nodes in under 20 ms. Additionally we should only be interested in the time it takes to render the entire page as this is where Search API will incur a penalty due to hitting the database.

3. What's your setup? Is you Solr hosted locally? Is you database hosted locally? Database hosted remotely again will penalize Search API as there's additionally latency in those queries and ApacheSolr wouldn't see these performance losses. Having Solr remote should penalize both modules the same I would assume.

4. Are you using an Entity Cache like the entitycache module? This is also important as it will let us determine Search API's speed as it will reduce the need to hit the database. Additionally with only 98 nodes, it would be assumed that all your entities are already pre-cached. This is perhaps a case where one could say testing with 10,000 nodes will make a difference as it's more likely that your nodes are not pre-cached. You should be testing Search API with entitycache's empty, as this is where an entity_load will need to perform the most amount of queries and incur the largest performance hit.

5. What caching backend are you using (Database (default), memcache or redis). This is important for determining the speed of the entity cache.

6. What hardware is this running on? I'm curious because I might have to purchase the same stuff, if I can render my uncached pages with Drupal in under 20 ms.

I finally went through the process of changing my sites most common apachesolr results, to avoid node_load_multiple() and node_view_multiple() and simply style what's returned back from apachesolr.

Here are my initial results after performing this change.

My site does not use page or block view, but relies heavily on memcache and entity cache backends instead. So these stats i refreshed the pages 3 times, then took the xhprof, so those caches would be primed.

This is from a live site. My solr index has ~45,000 entries and these queries are geo-sorts.

Homepage (16 node results, 5x solr queries):
http://i.imgur.com/7KwBJpGh.jpg
Before: 950ms (141k function calls)
After: 480ms (58k function calls)

Search Page (22 node results, 2x solr queries):
http://i.imgur.com/1Ka1RPjh.jpg
Before: 1,300 ms (162k function calls)
After: 304 ms (46k function calls)

This shows what happens when avoiding node_load() with regards to performance. My previous site was about as optimized as possible as well for my node_load_multiples and node_view_multiples, but as you can see, just not calling them at all is a huge win.

PS. COuldn't figure out how attach files with out updating the issue, so uploaded to imgur, which d.org doesn't let us link to it seems:
So here's the imgur album:
http://imgur.com/a/kZ5Xw

What is the conclusion please?
Is solr or search api faster search?

Many Thanks for your help

I would say, hitting solr with out ever needing to hit the database is faster. I would also say, avoiding views is faster.

Thus I would say, apachesolr module, personally.

With regards to what is faster when page & block cached are turned on, I could not tell you. I would still assume the solution which avoids the database and loading views code is faster.

I've heard https://drupal.org/project/mongodb is very good too.
But I haven't tried it yet