The exclude function was doing an unnecessary static cache clear when the loaded entity was actually in the memory. I moved some functions around and I think this will actually improve the speed of the indexing.

Please test and let me know the results.

Comments

janusman’s picture

Looks good on first glance!

nick_vh’s picture

With 500 items of d.o, and drush and the patch

nickveenhof@Nick-Veenhofs-MacBook-Pro:~/Sites/dev/drupalorg_d7/site/sites/all/modules/apachesolr$ time drush solr-index
Inspected 500 of 50481 entities. Submitted 500 documents to Solr [ok]
500 items successfully processed. 500 documents successfully sent to Solr. [status]

real 1m29.258s
user 1m0.704s
sys 0m2.059s

Without the patch

Inspected 500 of 49981 entities. Submitted 500 documents to Solr [ok]
500 items successfully processed. 500 documents successfully sent to Solr. [status]

real 2m0.622s
user 1m24.346s
sys 0m3.041s

I'll do 10 of those tests to see if this is actually a real improvement

janusman’s picture

Looks like a pretty good improvement if the remaining tests come out similar.

Functionally-wise, we should make extra sure things are indeed being removed from the static cache, and if so this is a no-brainer. Great work!

nick_vh’s picture

StatusFileSize
new99.07 KB

Here's some tests I did. It seems a better use of the entity cache gives us an average speedup of 114%, median (not super significant here) 115%.

The tests have been done on a D7 port of Drupal.org so the nodes were not generated. The procedure was the following

time drush solr-delete-index && time drush solr-mark-all && time drush solr-index

This means that we always index from scratch and thus do the same test over and over again. Each test was done 5 times to get a more realistic view.

Average time it took to index 500 entities without the patch :
158.941 seconds, which is 3.16 entities per second.
Average time it took to index 500 entities with the patch :
139.185 seconds, which is 3.59 entities per second.

Definitely an improvement! (please ignore the attachment, it selected the wrong column)

nick_vh’s picture

StatusFileSize
new80.09 KB

Screenshot_1_25_13_12_04.png

nick_vh’s picture

Status: Needs review » Fixed

No effect on D6, so marking as committed and fixed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.