AnnotatedClassDiscovery::getDefinitions takes 8-17 s (!!!) when viewing 50 nodes with up to 4 comments.

XHProf Link:

http://www.lionsad.de/xhprof-kit/xhprof/xhprof_html/?run=51f3790f24ac9&s...

Steps to reproduce

* Install Drupal 8
* drush -y dl devel
* drush -y en devel devel_generate
* Generate 50 nodes with up to 4 comments
* Change the front-page view to show:
** 50 nodes
** Change the row mode plugin to show comments (tick the checkbox)

* Load the frontpage and wait, wait, wait ...

Steps to measure

* Install xhprof-kit (https://github.com/LionsAd/xhprof-kit)
* Goto /index-perf.php
* Follow the xhprof-link at the bottom of the page
* Find the function and look yourself.

Proposed resolution

@todo

CommentFileSizeAuthor
#1 drupal-2051847-1.patch4.01 KBdawehner

Comments

dawehner’s picture

Status: Active » Needs review
StatusFileSize
new4.01 KB

This shows all kind of actual different kind of bugs.

fabianx’s picture

Status: Reviewed & tested by the community » Needs review

#1 fixes the performance problem and it is only taking 2s instead of 10s before and Annotations taking 8s before:

XHProf-Run: http://www.lionsad.de/xhprof-kit/xhprof/xhprof_html/?run=51f3c0cd6b1da&s...

=> I don't know what this is doing and why it is necessary (hence can't RTBC it), but it fixes the performance bug.

XHProf-Diff:

=== 8.x..performance-critical compared (51f4471ed94be..51f3c0cd6b1da):

ct  : 2,018,308|494,965|-1,523,343|-75.5%
wt  : 10,029,461|1,940,315|-8,089,146|-80.7%
cpu : 10,004,625|1,920,121|-8,084,504|-80.8%
mu  : 41,061,848|41,072,312|10,464|0.0%
pmu : 41,963,160|42,900,464|937,304|2.2%

http://www.lionsad.de/xhprof-kit/xhprof/xhprof_html/?run2=51f3c0cd6b1da&...

dawehner’s picture

The reason why this works is the following code in the plugin manager:

  public function getDefinition($plugin_id) {
    // Fetch definitions if they're not loaded yet.
    if (!isset($this->definitions)) {
      $this->getDefinitions();
    }
    // Avoid using a ternary that would create a copy of the array.
    if (isset($this->definitions[$plugin_id])) {
      return $this->definitions[$plugin_id];
    }
    return array();
  }

So instead of asking the discovery for every plugin definition, it builds them once and reuses it.

fabianx’s picture

Status: Needs review » Reviewed & tested by the community

This makes sense to me, tests still pass => RTBC

A quick grep for discovery->getDefinition() shows:

DiscoveryTestBase.php:    $this->assertEqual($this->discovery->getDefinitions(), $this->expectedDefinitions);
core/modules/system/lib/Drupal/system/Tests/Plugin/Discovery/DiscoveryTestBase.php:      $this->assertIdentical($this->discovery->getDefinition($id), $definition);
core/modules/system/lib/Drupal/system/Tests/Plugin/CacheDecoratorTest.php:    $definitions = $this->discovery->getDefinitions();
core/modules/system/lib/Drupal/system/Tests/Plugin/CacheDecoratorTest.php:    $definitions = $this->discovery->getDefinitions();
core/modules/system/lib/Drupal/system/Tests/Plugin/CacheDecoratorTest.php:    $this->discovery->getDefinitions();
core/modules/system/lib/Drupal/system/Tests/Plugin/CacheDecoratorTest.php:    $definition = $this->discovery->getDefinition('banana');
core/modules/system/lib/Drupal/system/Tests/Plugin/CacheDecoratorTest.php:    $definitions = $this->discovery->getDefinitions();
core/modules/field/lib/Drupal/field/Plugin/Type/Widget/WidgetFactory.php:    $plugin_definition = $this->discovery->getDefinition($plugin_id);
core/lib/Drupal/Core/Plugin/DefaultPluginManager.php:    $definitions = $this->discovery->getDefinitions();
core/lib/Drupal/Core/Plugin/Factory/ContainerFactory.php:    $plugin_definition = $this->discovery->getDefinition($plugin_id);
core/lib/Drupal/Component/Plugin/PluginManagerBase.php:    return $this->discovery->getDefinition($plugin_id);
core/lib/Drupal/Component/Plugin/PluginManagerBase.php:    return $this->discovery->getDefinitions();
core/lib/Drupal/Component/Plugin/Factory/ReflectionFactory.php:    $plugin_definition = $this->discovery->getDefinition($plugin_id);
core/lib/Drupal/Component/Plugin/Factory/DefaultFactory.php:    $plugin_definition = $this->discovery->getDefinition($plugin_id);

I am not sure if the factories need to use the discovery, but I trust dawehner having done the right thing above. (hence leaving RTBC)

tim.plunkett’s picture

Status: Needs review » Reviewed & tested by the community

In the Factory classes, using $this->discovery is correct
And the test ones are fine as is.
+1 for RTBC

Anonymous’s picture

Status: Reviewed & tested by the community » Needs work

so, we're happy with Yet Another Static Cache one step removed from whatever cache invalidation mechanism is in place for discovery?

i get it's only within a single request, but perhaps we should push it down a level?

tim.plunkett’s picture

Status: Needs work » Reviewed & tested by the community

We're not adding any of that here. Just not bypassing it by calling into a deeper object.

berdir’s picture

Yes, this changed by using the default plugin manager that does caching itself and not as part of the discovery decorator chain.

Fix is correct and is something we need to check for upcoming conversions.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Nice find Fabianx!

Committed 8242143 and pushed to 8.x. Thanks!

catch’s picture

+++ b/core/lib/Drupal/Core/Archiver/ArchiverManager.phpundefined
@@ -40,7 +40,7 @@ public function __construct(\Traversable $namespaces, CacheBackendInterface $cac
   public function createInstance($plugin_id, array $configuration = array()) {
-    $plugin_definition = $this->discovery->getDefinition($plugin_id);

It's not good that there's two ways to get exactly the same information, one of which should never be used. Could we discuss making it harder to make this mistake?

Also looking at the xhprof we still have menu local tasks taking 20ms, I followed up on #2046565: Cache the local action plugins that appear per route.

dawehner’s picture

The reason why this was in core all over the place is that the plugin manager got mostly replaced by the DefaultPluginManager which handles the caching instead of the decorator approach of before. $this->discovery has been good before (as it was the cache decorator),
so we probably failed to review the change properly.

berdir’s picture

I guess the question is if we need a getDefinition($id) on the discovery, or if the discovery should only be responsible for returning an array of all definitions. Then it would be much more obvious to use $this->getDefinition() instead of $this->discovery->getDefinition(). That would obviously be an API change, so maybe just document it properly..

catch’s picture

I'd be OK with an API change if it stops this.

yched’s picture

Agreed with #12 that there shouldn't be a way to ask the discovery about a single definition, this should be the manager's task, discovery now is only about finding all.

That would mean removing getDefinition() from DiscoveryInterface (and from existing discovery classes / decorators ?), and adding it to PluginManagerInterface (that would be the only explicit method in there, PluginManagerInterface currently only extends other interfaces...)

berdir’s picture

neclimdul’s picture

Skimming the patch, all the changes seem to be to managers that are hard coding logic they copied out of factories instead of using factories. I don't understand why that's being done as I don't see any specific logic in any of them to justify it.

Automatically closed -- issue fixed for 2 weeks with no activity.