Problem/Motivation
While doing a performance test, I loaded 1000 entities using {EntityType}::loadMultiple
, and the execution was really slow, 15 secs. After some debugging, I found that the slowness is because of ContentEntityStorageBase::setPersistentCache
, it contributed roughly 14 secs to the load time.
The code goes through each loaded entities and sets the cache for them one-by-one.
If the code collects all the items first and then call the cache backend only once (setMultiple
), the cache setting time decreases below 1 sec and the overall load time of entities to 1.5 secs.
The specific call stack:
- EntityBase::loadMultiple
- EntityStorageBase::loadMultiple
- SqlContentEntityStorage::doLoadMultiple
- SqlContentEntityStorage::setPersistentCache
- ContentEntityStorage::setPersistentCache
- DatabaseBackend::set
Steps to reproduce
- Use
{EntityType}::loadMultiple
on high number of entities (say, 1000) - Measure times at the above mentioned calls before applying the patch, e.g.:
$start = microtime(true); // the function call kint(microtime(true) - $start);
- Apply the patch, measure again
Note: My installation is Drupal 8.9.2, I tested with that, but created the patch based on 9.0.x (dev). The same patch applies to 8.9.2 as well.
Proposed resolution
Collect all items to be cached first in an array, then call {cache backend}->setMultiple
.
Remaining tasks
- Review the attached patch.
- As the items to be cached are first collected in an array, the memory consumption can slighty increase. Should not be a big issue, but please review this and comment on it.
User interface changes
-
API changes
-
Data model changes
-
Release notes snippet
Later, first review the proposal and check if it causes any regression.
Comment | File | Size | Author |
---|---|---|---|
#16 | 3164428-16-9.4.x.patch | 1.99 KB | longwave |
#16 | 3164428-16.patch | 1.97 KB | longwave |
| |||
#15 | interdiff_13-15.txt | 1.03 KB | sahil.goyal |
#15 | 3164428-15.patch | 1.75 KB | sahil.goyal |
#13 | interdiff-3164428-12_13.txt | 1.06 KB | Anchal_gupta |
Comments
Comment #2
DonAtt CreditAttribution: DonAtt as a volunteer commentedComment #3
DonAtt CreditAttribution: DonAtt as a volunteer commentedComment #4
DonAtt CreditAttribution: DonAtt as a volunteer commentedOne test case fails because it expects the cache's
set
method to be invoked - which was replaced bysetMultiple
.All other tests passed.
Comment #5
DonAtt CreditAttribution: DonAtt as a volunteer commentedComment #6
DonAtt CreditAttribution: DonAtt as a volunteer commentedComment #10
longwaveNice find! I've just found the exact same problem. I have a controller that loads many entities and New Relic traces that show me that approx 70% of the page load time (up to 7 seconds in some cases) is spent in
\Drupal\Core\Entity\ContentEntityStorageBase::setPersistentCache()
.#5 solves the problem for me and still applies to 9.4.x.
Comment #11
alexpottNice idea! Needs a reroll...
Comment #12
Ankit.Gupta CreditAttribution: Ankit.Gupta at Dotsquares Ltd. commentedReroll the patch #5 with Drupal 9.4.x
Comment #13
Anchal_gupta CreditAttribution: Anchal_gupta at Srijan | A Material+ Company for Drupal India Association commentedI have fixed cs error.
Comment #14
alexpottThis the default - so we don't need it. The only reason it was in the set was because we needed to pass in the tags. See \Drupal\Core\Cache\CacheBackendInterface::setMultiple()
Comment #15
sahil.goyal CreditAttribution: sahil.goyal as a volunteer and at Srijan | A Material+ Company for Drupal India Association commentedAs per the #14 found that the expire key is default, so i have uploading the patch for 9.4.x-dev as per suggestion and attaching interdiff, Thanks..
Comment #16
longwaveFixed unused use statement, cleaned up coding style a bit in the test. We also need a slightly different patch for 9.4.x if we are to backport this.
Comment #17
sahil.goyal CreditAttribution: sahil.goyal as a volunteer and at Srijan | A Material+ Company for Drupal India Association commentedYeah @longwave, now it looks good.. Thanks.
Comment #18
init90Comment #20
catchCommitted/pushed/cherry-picked the respective patches to 10.1.x back through to 9.4.x, thanks!
Comment #23
cilefen CreditAttribution: cilefen commentedThere is a report of a performance regression: #3327118: Chunk multiple cache sets into groups of 100 to avoid OOM/max_allowed_packet issues.