Problem/Motivation

While doing a performance test, I loaded 1000 entities using {EntityType}::loadMultiple, and the execution was really slow, 15 secs. After some debugging, I found that the slowness is because of ContentEntityStorageBase::setPersistentCache, it contributed roughly 14 secs to the load time.

The code goes through each loaded entities and sets the cache for them one-by-one.
If the code collects all the items first and then call the cache backend only once (setMultiple), the cache setting time decreases below 1 sec and the overall load time of entities to 1.5 secs.

The specific call stack:

  1. EntityBase::loadMultiple
  2. EntityStorageBase::loadMultiple
  3. SqlContentEntityStorage::doLoadMultiple
  4. SqlContentEntityStorage::setPersistentCache
  5. ContentEntityStorage::setPersistentCache
  6. DatabaseBackend::set

Steps to reproduce

  • Use {EntityType}::loadMultiple on high number of entities (say, 1000)
  • Measure times at the above mentioned calls before applying the patch, e.g.:
    $start = microtime(true);
    // the function call
    kint(microtime(true) - $start);
    
  • Apply the patch, measure again

Note: My installation is Drupal 8.9.2, I tested with that, but created the patch based on 9.0.x (dev). The same patch applies to 8.9.2 as well.

Proposed resolution

Collect all items to be cached first in an array, then call {cache backend}->setMultiple.

Remaining tasks

  • Review the attached patch.
  • As the items to be cached are first collected in an array, the memory consumption can slighty increase. Should not be a big issue, but please review this and comment on it.

User interface changes

-

API changes

-

Data model changes

-

Release notes snippet

Later, first review the proposal and check if it causes any regression.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

DonAtt created an issue. See original summary.

DonAtt’s picture

DonAtt’s picture

Status: Active » Needs review
DonAtt’s picture

One test case fails because it expects the cache's set method to be invoked - which was replaced by setMultiple.

public function testLoadMultiplePersistentCacheMiss() {
// ...
$this->cache->expects($this->once())
      ->method('set')
      ->with($key, $entity, CacheBackendInterface::CACHE_PERMANENT, [$this->entityTypeId . '_values', 'entity_field_info']);
// ...

All other tests passed.

DonAtt’s picture

DonAtt’s picture

Issue tags: +Performance

Version: 9.0.x-dev » 9.1.x-dev

Drupal 9.0.10 was released on December 3, 2020 and is the final full bugfix release for the Drupal 9.0.x series. Drupal 9.0.x will not receive any further development aside from security fixes. Sites should update to Drupal 9.1.0 to continue receiving regular bugfixes.

Drupal-9-only bug reports should be targeted for the 9.1.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.2.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.1.x-dev » 9.3.x-dev

Drupal 9.1.10 (June 4, 2021) and Drupal 9.2.10 (November 24, 2021) were the last bugfix releases of those minor version series. Drupal 9 bug reports should be targeted for the 9.3.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

longwave’s picture

Title: Consider cacheBackend->setMultiple in ContentEntityStorageBase::setPersistentCache » Use cacheBackend->setMultiple in ContentEntityStorageBase::setPersistentCache
Category: Feature request » Task
Status: Needs review » Reviewed & tested by the community

Nice find! I've just found the exact same problem. I have a controller that loads many entities and New Relic traces that show me that approx 70% of the page load time (up to 7 seconds in some cases) is spent in \Drupal\Core\Entity\ContentEntityStorageBase::setPersistentCache().

#5 solves the problem for me and still applies to 9.4.x.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs reroll

Nice idea! Needs a reroll...

Ankit.Gupta’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
1.83 KB

Reroll the patch #5 with Drupal 9.4.x

Anchal_gupta’s picture

I have fixed cs error.

alexpott’s picture

Status: Needs review » Needs work
+++ b/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php
@@ -1113,9 +1113,15 @@ protected function setPersistentCache($entities) {
+        'expire' => CacheBackendInterface::CACHE_PERMANENT,

This the default - so we don't need it. The only reason it was in the set was because we needed to pass in the tags. See \Drupal\Core\Cache\CacheBackendInterface::setMultiple()

sahil.goyal’s picture

As per the #14 found that the expire key is default, so i have uploading the patch for 9.4.x-dev as per suggestion and attaching interdiff, Thanks..

longwave’s picture

Fixed unused use statement, cleaned up coding style a bit in the test. We also need a slightly different patch for 9.4.x if we are to backport this.

sahil.goyal’s picture

Yeah @longwave, now it looks good.. Thanks.

init90’s picture

Status: Needs review » Reviewed & tested by the community

  • catch committed 226e6c4 on 10.0.x
    Issue #3164428 by DonAtt, longwave, sahil.goyal, Anchal_gupta, alexpott...
  • catch committed 7b9edd6 on 10.1.x
    Issue #3164428 by DonAtt, longwave, sahil.goyal, Anchal_gupta, alexpott...
  • catch committed 274f8d9 on 9.5.x
    Issue #3164428 by DonAtt, longwave, sahil.goyal, Anchal_gupta, alexpott...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed/pushed/cherry-picked the respective patches to 10.1.x back through to 9.4.x, thanks!

  • catch committed c00e950 on 9.4.x
    Issue #3164428 by DonAtt, longwave, sahil.goyal, Anchal_gupta, alexpott...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

cilefen’s picture