Problem/Motivation

Follow up to #2203411: Convert drupal_get_library() into a service, currently library info is cached per extension. So, for example, a vanilla D8 install, front page, there are ~ 10 calls to different library extension cache items.

Proposed resolution

Consolidate the per extension caches into one item, which is loaded by the Library service. We can still use getLibraryByExtension, hence no api changes, the class just gets its cached data once.

Remaining tasks

Wait for #2203411: Convert drupal_get_library() into a service, Decide best approach for getting all extension library info, implement above, tests.

User interface changes

None

API changes

None

Comments

damiankloip’s picture

Issue summary: View changes
damiankloip’s picture

Priority: Normal » Major
catch’s picture

Since this is by extension, we'll need to have the active theme in the cache key, none of these should be requested before the theme is set so should be OK.

Also assuming this is for all discovered libraries. Some might not be used, or only on rarely visited admin pages etc. so might be worth CacheCollector here?

dawehner’s picture

Yeah we should come up with a scenario and have a look at the needed amount of memory for it.

sun’s picture

Not only the active, but all enabled themes.

The memory consumption should be small. Even smaller if we'd find a way to omit the default/empty properties (e.g., by caching the original/unprocessed data instead of the processed data).

catch’s picture

Postponing this on #2228291: Move all plugin and other discovery data into a cache_discovery bin too. With that and the PhpStorage cache backend we might have come up with a way to avoid the i/o memory trade-off.

damiankloip’s picture

Status: Postponed » Active
damiankloip’s picture

Status: Active » Postponed

Whoops, sorry!

catch’s picture

Status: Postponed » Closed (duplicate)

Moving this to duplicate. If the local cache stuff turns out to be a dead end, we can re-open this.

damiankloip’s picture

Status: Closed (duplicate) » Needs review

Seems still beneficial to do this if we can? This will not yield much more data than some plugin definitions? Sorry, changing back to needs review so it gets your attention again :)

catch’s picture

Yeah I don't mind doing this additionally to the other one. It's a lot less complex than config pre-loading.

damiankloip’s picture

StatusFileSize
new21.18 KB

Here is an initial patch - not finished. Just didn't want to get too far with converting/adding tests until we get some consensus on the approach. So this will fail.

I think the best way of doing this is to use the cache collector. We can then just rely on trying to get what's requested, like now, rather than trying to discover everything up front. I have moved basically everything out of the LibraryDiscovery class, and moved it to a LibraryDiscoveryCollector class. This extends CacheCollector and aggregates all of the library info extension data. From quick tests on the front page, this patch made 12 less get() queries to the cache backend.

I am not sure about what I have done in this patch as all the library discovery logic lives in the new collector class. Which is probably not a good idea.

Anyway, this is a start we can start picking/criticising etc...

Status: Needs review » Needs work

The last submitted patch, 12: 2223143.patch, failed testing.

damiankloip’s picture

Status: Needs work » Needs review
StatusFileSize
new21.48 KB
new16.08 KB

Ok, added a new LibraryDiscoveryParser service. This gets injected into the LibraryDiscoveryCollector instead now. This separates the logic between simple cache collector implementation and the parsing logic to get library data for an extension.

Will fix tests once I get some validation/happiness with the approach.

Status: Needs review » Needs work

The last submitted patch, 14: 2223143-14.patch, failed testing.

dawehner’s picture

+++ b/core/lib/Drupal/Core/Asset/LibraryDiscoveryCollector.php
@@ -0,0 +1,75 @@
+/**
+ * Class LibraryDiscoveryCollector.
+ */
+class LibraryDiscoveryCollector extends CacheCollector {

Shit really? In general I really wonder whether it is worth to split this up into 3 classes tbh. As long you change the tests properly this though should work fine.

damiankloip’s picture

Status: Needs work » Needs review
StatusFileSize
new39.98 KB
new19.47 KB

Converted LibraryDiscoveryTest to LibraryDiscoveryParserTest, added a new test for the LibraryDiscoveryCollector.

damiankloip’s picture

17: 2223143-17.patch queued for re-testing.

dawehner’s picture

+++ b/core/lib/Drupal/Core/Asset/LibraryDiscoveryCollector.php
@@ -0,0 +1,75 @@
+ * Class LibraryDiscoveryCollector.
...
+class LibraryDiscoveryCollector extends CacheCollector {

at least the doc does not tell you anything wrong :P

damiankloip’s picture

StatusFileSize
new40.02 KB
new529 bytes

Good point!

sun’s picture

Issue summary: View changes
  1. +++ b/core/core.services.yml
    @@ -810,7 +810,15 @@ services:
    +  library.discovery.parser:
    ...
    +  library.discovery.collector:
    

    Let's order these two alphabetically.

  2. +++ b/core/lib/Drupal/Core/Asset/LibraryDiscoveryCollector.php
    @@ -0,0 +1,75 @@
    +    $this->storage[$key] = $this->discoveryParser->buildLibrariesByExtension($key);
    

    "buildLibrariesByExtension" is very long.

    $this->discoveryParser->getByExtension($name) ?

    Is there actually a different way than by extension, or do we intend to add one in the future?

    If not, just LibraryParser->get($extension)?

  3. +++ b/core/lib/Drupal/Core/Asset/LibraryDiscoveryParser.php
    @@ -43,85 +33,14 @@ class LibraryDiscovery implements LibraryDiscoveryInterface {
    +   *   The moduel handler.
    

    Typo in "moduel"

  4. +++ b/core/lib/Drupal/Core/Asset/LibraryDiscoveryParser.php
    @@ -135,8 +54,13 @@ protected function setCache($extension, $information) {
    +  public function buildLibrariesByExtension($extension) {
    +    if (isset($this->libraries[$extension])) {
    +      return $this->libraries[$extension];
    +    }
    

    Hm. If we're going to split it all up this radically, then I wonder why there is still a separate/custom/duplicate property cache on the parser?

    Shouldn't the parser just parse when told to do so?

  5. +++ b/core/lib/Drupal/Core/Asset/LibraryDiscoveryParser.php
    @@ -309,4 +220,18 @@ protected function parseLibraryInfo($extension, $library_file) {
    +  protected function drupalGetPath($type, $name) {
    

    Btw, we still need to inject some $type knowledge into all these classes...

    Two possible approaches:

    A. Use the legacy $type, $name pattern of drupal_get_path() & Co.

    B. Use Extension objects. (they have ->getType() built-in)

damiankloip’s picture

StatusFileSize
new43.88 KB
new16.01 KB

Thanks for the review.

1. Fixed
2. I thought you added this originally ?! :p I think the buildByExtension option is good?
3. Fixed
4. Fixed, this was just a precaution more than anything. As discussed, as the caching layer worries about this now. This is just extra memory usage.
5. That's a valid concern. It is pre existing though, and if we can I would like to punt that to a follow up. This patch is big enough already. I don't think this issue would make that more difficult. If we needed to we could change the parameters of the parser and collector classes. Either way we would need to change the LibraryDiscovery interface parameters to achieve this (which is why I would like to isolate that discussion after this).

If you are happy, I will create a follow up for that so we can start discussing there?

damiankloip’s picture

Issue summary: View changes
sun’s picture

Status: Needs review » Reviewed & tested by the community

Thanks!

Yeah, let's definitely create a follow-up for the extension type issue - that magic type detection hack was definitely meant to be temporary only...

It would also be great to create a separate follow-up issue to (1) rename the service IDs into 'asset.library_discovery*' and optionally (2) move the classes into a \Asset\Discovery subnamespace and remove "Discovery" from the class names.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed c5942ee and pushed to 8.x. Thanks!

  • Commit c5942ee on 8.x by alexpott:
    Issue #2223143 by damiankloip: Consolidate library extension caches into...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

penyaskito’s picture