Problem/Motivation

Doing some profiling...

CommentManager::getAllFields() is called in comment_node_links_alter(), if you display 100 nodes then it is called 100 times, which takes 3.5ms/1.3% of the page request time in my case. Considering that calling EntityManager::getFieldMap(), which is statically cached, only takes 260microseconds (7%), that's how fast it would be if we'd have a static cache directly in that method.

Proposed resolution

Build another statically cached map in entity manager keyed by field type.

Remaining tasks

review, profile, commit

User interface changes

no

API changes

Added EntityManagerInterface::getFieldMapByFieldType()
Removed CommentManagerInterface::getAllFields()

Original report by @Berdir

Doing some profiling...

CommentManager::getAllFields() is called in comment_node_links_alter(), if you display 100 nodes then it is called 100 times, which takes 3.5ms/1.3% of the page request time in my case. Considering that calling EntityManager::getFieldMap(), which is statically cached, only takes 260microseconds (7%), that's how fast it would be if we'd have a static cache directly in that method.

Comments

berdir’s picture

Issue tags: +Novice
larowlan’s picture

StatusFileSize
new1.84 KB

Here's a failing test to get you started, your challenge, should you choose to accept it - is to make this patch green.

andypost’s picture

Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch, 2: comment-mgr-all-fields-2269025.fail_.patch, failed testing.

grom358’s picture

Assigned: Unassigned » grom358
Status: Needs work » Needs review
StatusFileSize
new3.67 KB

The potential issue with this patch is if EntityManagerInterface::clearCachedFieldDefinitions() is called during a request, it doesn't result in cache clear for CommentManager::getAllFields().

larowlan’s picture

Status: Needs review » Reviewed & tested by the community

Looks good to me

berdir’s picture

Yeah, almos too god, actually :)

+++ b/core/modules/comment/lib/Drupal/comment/CommentManager.php
@@ -87,17 +97,18 @@ public function getFields($entity_type_id) {
+    if (!$this->fieldMap) {
+      $map = $this->entityManager->getFieldMap();

This means that we can't separate between not being executed yet and an empty array.

Wondering if it's worth to optimize and differ between not-yet-executed and executed-but-no-fields-were-found, which means that we'd set the initial value to NULL and check with isset().

I suspect that is the reason this is not failing right now, because all tests just set up one field and we re-run this until we actually have a field and then don't need to worry about the static cache going stale?

Should we extend one of the web tests to delete the field and then check that th list was updated I suspect it would return stale information and we'd need to introduce a way to reset the fields.. that's always the downside of adding more caching :)

larowlan’s picture

Status: Reviewed & tested by the community » Needs work

So what if we added a reset method and then some timely hooks to fire to call the reset on update/delete/insert of field_config entities of type comment?

grom358’s picture

Or on EntityManagerInterface have getFieldMapByType() method or getFieldMap($type) or something like that, and have EntityManager cache the filtered fieldMap for CommentManager and when fieldMap cache is cleared with clearCachedFieldDefinitions() is also clears the filtered fieldMap.

grom358’s picture

Status: Needs work » Needs review
StatusFileSize
new6.32 KB

Patch using EntityManager to handle the caching, so its refreshed when fieldMap is.

andypost’s picture

Issue tags: +API addition, +API clean-up

Makes sense! Let's just move CM method to EM!

Suppose issue should change title and summary.

  1. +++ b/core/modules/comment/lib/Drupal/comment/CommentManager.php
    @@ -52,6 +52,16 @@ class CommentManager implements CommentManagerInterface {
    +  protected $fieldMap = array();
    

    remains?

  2. +++ b/core/modules/comment/lib/Drupal/comment/CommentManager.php
    @@ -87,17 +97,7 @@ public function getFields($entity_type_id) {
       public function getAllFields() {
    ...
    -    return $comment_fields;
    +    return $this->entityManager->getFieldMapByFieldType('comment');
    

    there's no sense to handle this method once EM does the same

  3. +++ b/core/modules/comment/tests/Drupal/comment/Tests/CommentManagerTest.php
    @@ -0,0 +1,64 @@
    +class CommentManagerTest extends UnitTestCase {
    

    this should test new entity manager method

mikeegoulding’s picture

New drupal contributor here, Mike Goulding, I'm at the Drupalcon Austin sprint. I noticed this needs to be re-rolled and I'm going to take care of that now.

mikeegoulding’s picture

StatusFileSize
new5.97 KB

Rerolled patch from comment #10.

Status: Needs review » Needs work

The last submitted patch, 13: comment-mgr-all-fields-2269025-3.patch, failed testing.

moshe weitzman’s picture

This has once again shown up as a performance problem in HEAD. If anyone is available to give this a final push, you will be kissed by the moon and stars.

penyaskito’s picture

Assigned: grom358 » penyaskito

Working on it.

penyaskito’s picture

Status: Needs work » Needs review
StatusFileSize
new5.81 KB
new8.84 KB

I started back from last green (#10). I don't fully understand what is asked in #11.

andypost’s picture

Issue summary: View changes
Issue tags: -Novice +needs profiling
StatusFileSize
new4.35 KB
new6.9 KB

Let's remove unused method now from comment manager

Status: Needs review » Needs work

The last submitted patch, 18: comment-mgr-2269025-18.patch, failed testing.

penyaskito’s picture

Status: Needs work » Needs review
StatusFileSize
new7.26 KB
new611 bytes

$this->discovery does not exist.

Status: Needs review » Needs work

The last submitted patch, 20: 2269025-20.patch, failed testing.

penyaskito queued 20: 2269025-20.patch for re-testing.

The last submitted patch, 20: 2269025-20.patch, failed testing.

The last submitted patch, 18: comment-mgr-2269025-18.patch, failed testing.

penyaskito’s picture

Please ignore #20.
$this->discovery is an attribute inherited in the parent class, and theorically is correctly initialized from constructor.
I cannot find what is wrong there.

andypost’s picture

@penyaskito I think we need to extend existing EM unit test, no reason to test CM because EM gets new method

penyaskito’s picture

@andypost But the failure comes from BreakpointMediaQueryTest :-S

berdir’s picture

Nope, the failing test is Drupal\comment\Tests\CommentManagerTest->testGetFields, cd core; ./vendor/bin/phpunit to see the backtrace (if you have xdebug enabled).

That test mocks EntityManager without calling the constructor, that's why it's not there. And yes, it doesn't make sense to test this method in CommentManagerTest.

andypost’s picture

Status: Needs work » Needs review
StatusFileSize
new2.75 KB
new7.34 KB

Fixed CM test, still needs to extend core/modules/comment/tests/src/CommentManagerTest.php

moshe weitzman’s picture

Priority: Normal » Major

Increasing priority due to performance

andypost’s picture

have no time now, the issue needs extend entity manager test to cover new method

grom358’s picture

Yeah i'm been too busy with other things to come back to this. So all that is left is to add tests to EntityManagerTest ?

yched’s picture

+1 on the approach, never really liked CM::getAllFields() :-).

Just a minor note that the phpdoc for the new EM property is a bit weird.

yched’s picture

Also, there's probably some code in file.module that could leverage that new feature ?

penyaskito’s picture

StatusFileSize
new4.21 KB

Is this the way to go for the test? I'm not sure if I understood correctly.

andypost’s picture

@penyaskito yes, it is!
Also needs to check #35

penyaskito’s picture

Added test at #36 as proper patch. Small addition of some asserts.
Changed the phpdoc as @yched suggested in #34, not sure if this is more convincing.

Checked file.module for extending the usage of the new methods in there as @yched suggests in #35, but couldn't find any case.

berdir’s picture

+++ b/core/lib/Drupal/Core/Entity/EntityManagerInterface.php
@@ -92,6 +92,20 @@ public function getFieldStorageDefinitions($entity_type_id);
+   * Collects a lightweight map of fields across bundles filtered by field type.

I don't think "Collect" here makes sense? It might do that internally, but that's an implementation detail that is not relevant for the API/Interface. So just, "Returns"?

I'm not sure about CommentManagerTest. It's not a unit test of CommentManager, it's mostly testing the static cache in EntityManager now, for which we now have a dedicated unit test. IMHO, we should either simplify that to mock getFieldMapByFieldType() and just verify that we call that method or remove it.

AFAIK, the only other call to getFieldMap() that we could update is is in Vocabulary::postSave()

penyaskito’s picture

Status: Needs review » Needs work
penyaskito’s picture

Status: Needs work » Needs review
StatusFileSize
new12.63 KB
new3.43 KB

Changed phpdoc in EntityManagerInterface.
Changed CommentManagerTest for mocking getFieldMapByFieldType().
Reused getFieldMapByFieldType() in Vocabulary::postSave().

penyaskito’s picture

Oops, fixed indentation.

berdir’s picture

+++ b/core/modules/comment/tests/src/CommentManagerTest.php
@@ -34,19 +34,16 @@ public function testGetFields() {
     $entity_manager = $this->getMockBuilder('Drupal\Core\Entity\EntityManager')
       ->disableOriginalConstructor()
-      ->setMethods(array('getFieldMap', 'getDefinition'))
+      ->setMethods(array('getFieldMapByFieldType', 'getDefinition'))
       ->getMock();

You can probably simplify this to just getMock('...EntityManagerInterface') because we no longer have to call actual methods but just the one mocked method?

penyaskito’s picture

@Berdir is right.

berdir’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -needs profiling
StatusFileSize
new12.21 KB
new911 bytes

Unit test is very long, removed a few distracting lines that are not necessary for this, was considering to mock getFieldMap() so that we can just the specific method, but that would be way too much to still be able to RTBC this.

I don't think this needs specific profiling, improvements should be pretty obvious.

catch’s picture

Status: Reviewed & tested by the community » Fixed

Nice find. Committed/pushed to 8.x, thanks!

  • catch committed 381b05e on 8.0.x
    Issue #2269025 by penyaskito, andypost, grom358, Berdir, mikeegoulding,...
penyaskito’s picture

Kissed by the moon and the stars, thanks!

penyaskito’s picture

Assigned: penyaskito » Unassigned

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.