Problem/Motivation

The \Drupal\Core\Extension\ExtensionDiscovery class documentation implies that you can control whether or not extensions underneath a profile directory will be scanned or not. But this is not actually true, unless you fake it.

There are two ways to set the $profileDirectories member variable explicitly. One is in the constructor, and another is by calling the setProfileDirectories() method. The documentation for the member variable, constructor param, and method say:

  • Member variable: List of installation profile directories to additionally scan.
  • Method: Sets explicit profile directories to scan. // A list of installation profile directory paths relative to the system root directory (without trailing slash) to search for extensions.
  • Constructor: The available profile directories

All of these imply (at least to me) that if you set this to an empty directory, no profile directories should be scanned for modules.

However, what actually happens if you set this to an empty directory is that all profile directories are scanned.

What you can actually do in order to forbid profile directories from being scanned for modules is set it to something like ['foo'], so that it will only scan the non-existent "foo" profile. However, if you do that when you are scanning for profiles themselves, nothing will be returned.

Proposed resolution

Either:
a) Fix the documentation so it matches the behavior.
or
b) Fix the code so that if you pass in an empty directory (as opposed to NULL), no profile directories are scanned for modules/themes.

Decided on (a). See comment #7 for why (b) seemed too dangerous.

Remaining tasks

Decide and patch and test.

User interface changes

None.

API changes

None. Patch contains documentation and local/protected variable name changes only.

Data model changes

None.

Release notes snippet

None.

Comments

jhodgdon created an issue. See original summary.

andypost’s picture

There's few issues like related that trying to add white/blacklist to discovery but not sure that could help here

jhodgdon’s picture

I guess you could blacklist "profiles" and that would probably work, if the blacklist was settable.

larowlan’s picture

I think changing the empty vs NULL behaviour would possibly break things, because of PHPs loose typing.

At least one profile has to be scanned in runtime code, so if this is an issue in testing, I have no issue with using a dummy profile for that purpose

jhodgdon’s picture

Status: Active » Needs review
StatusFileSize
new14.22 KB

Yes, indeed. I didn't mention in my report, but I had already tried on my local machine to change this empty() to isset() in ExtensionDiscovery:

protected function filterByProfileDirectories(array $all_files) {
    if (empty($this->profileDirectories)) {
      return $all_files;
    }

The first test I tried running (the test I was actually working on) failed to even get through installation because it couldn't find the testing profile. It basically broke testing.

So... I looked into this (fairly messy!!!!) code in ExtensionDiscovery some more. I realized that it was tacitly assuming during scan() that if you were looking for profiles, then profileDirectories() must have been set to NULL. This is not necessarily the case -- it is never actually verified. That assumption also means that if you created an ExtensionDiscovery object and scanned for modules, then later scanned for profiles, it would most likely break.

So, I went through and cleaned up the code and documentation, and fixed that assumption. I ran one test locally to verify that this patch doesn't completely break the testing system... let's see what happens in the full test suite.

Status: Needs review » Needs work

The last submitted patch, 5: 3076098-5.patch, failed testing. View results

jhodgdon’s picture

Status: Needs work » Needs review
StatusFileSize
new14.6 KB

Well. I looked into the test fails. We have several places in Core where we were doing things like this:

    $listing = new ExtensionDiscovery($this->root);
    $listing->setProfileDirectories([]);

before doing a scan. What this does:
- If you were scanning for profiles, it has no effect, so it would still do what it always did, and return all profiles (see #3076329: ExtensionDiscovery returns testing profiles even when you ask it not to, which is about that buggy behavior).
- If you were scanning for modules, previously this would prevent the automatic filtering to just the active profile, and now it does what I think it should be doing, which is filtering so that no modules in any profile directories are being scanned.

As a side effect, ModuleExtensionList is able to find modules in non-active profiles. Which I think is actually a bug... But some of the test fails are from expecting this behavior. Others are from the TestDiscovery process making that same assumption. So... I think that it is too dangerous at this point to fix the actual bug, and what we should do to "solve" this problem is document the buggy behavior and how to get around it. So, Here's a new patch. I didn't bother with an interdiff because I really had to start over.

This is just a documentation patch, plus changing names of some local and protected variables from "files" to "extensions" where appropriate. And in one case, in a test that was looking into that protected member variable that I changed the name of.

jhodgdon’s picture

Issue summary: View changes

Updating issue summary.

andypost’s picture

Changing "files" to "extensions" looks great, because it is what it actually stores, but not sure about scope
Would be great to get more eyes on it... scope creep

I recall inability to find extensions in profiles caused to move all testing ones to core/profiles
Maybe it needs some test added here to illustrate?

PS: tests are often do manual discovery (or do copy #2850875: Dependency on configuration provided by install profile is throwing exception) so makes sense to create follow-up to "normalize" discovery inside profiles

jhodgdon’s picture

Title: Impossible to tell ExtensionDiscovery to not scan profile directories » ExtensionDiscovery documentation is wrong about scanning profile directories

This issue only ended up doing documentation and local variable name changes. I didn't change the code at all. I decided the scope creep was probably OK because there is no code change, just documentation cleanup... updating title

The code is currently perfectly capable of discovering profiles anywhere it can find modules. If you look at ExtensionDiscovery, it looks everywhere for all extensions, and then categorizes them by type after reading the .info.yml file to find out what type they are. This is necessary, because some test modules (for example) are not under tests/modules but just under tests, so you cannot infer the type unless you look at the .info file. Anyway regarding that see #3076329: ExtensionDiscovery returns testing profiles even when you ask it not to, which moved the profiles to a different location.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

daffie’s picture

Status: Needs review » Needs work

Thank you @jhodgdon for creating this patch.

I very much appreciate the extra documentation that this patch provides.
Especially where the working of this class is different for profiles.
Also renaming $files to $extensions is an improvement.

I have just one remark:

+++ b/core/lib/Drupal/Core/Extension/ExtensionDiscovery.php
@@ -57,16 +59,28 @@ class ExtensionDiscovery {
-   * Previously discovered files keyed by origin directory and extension type.
+   * Previously discovered extensions.
+   *
+   * The array keys are:
+   * - Site root directory
+   * - Search directory
+   * - Whether test directories were being scanned
+   * - Type of extension
+   * - Absolute path to the extension
+   *
+   * The array values at that level are \Drupal\Core\Extension\Extension[]
+   * objects.

How are the extensions now exactly keyed? This bit is confusing to me.

jhodgdon’s picture

That was meant to convey that it is a very deep array. At the top level, the array key is the site root directory. Then at the next level, the search directory. And so on.

I'm not sure how that could be worded better, any suggestions?

daffie’s picture

Can we change the line: "The array keys are" to "The array is keyed 5 layers deep. The array keys are from top to bottom:". If you can live with that change then it is for me RTBC.

jhodgdon’s picture

Status: Needs work » Needs review
StatusFileSize
new14.63 KB
new838 bytes

Hm... Maybe something like this?

daffie’s picture

Status: Needs review » Reviewed & tested by the community

Hm... Maybe something like this?

Your changes are also great. A multidimensional array with 5 layers is not very usual. So make it very clear that that is what we have here.
With the latest change it all look good to me.
For me it is RTBC.
Thank you @jhodgdon.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/lib/Drupal/Core/Extension/ExtensionDiscovery.php
@@ -123,20 +139,14 @@ public function __construct($root, $use_file_cache = TRUE, $profile_directories
    * The following directories will be searched (in the order stated):
    * - the core directory; i.e., /core
-   * - the installation profile directory; e.g., /core/profiles/standard
    * - the legacy site-wide directory; i.e., /sites/all
    * - the site-wide directory; i.e., /
    * - the site-specific directory; e.g., /sites/example.com
-   *
-   * To also find test modules, add
-   * @code
-   * $settings['extension_discovery_scan_tests'] = TRUE;
-   * @endcode
-   * to your settings.php.
+   * - the installation profile directory; e.g., /core/profiles/standard
    *

This is not correct as far as I can see.

The specific directory listed here core/profiles/standard is actually scanned when we scan /core. i.e. the first thing on the list. Contributed and custom profiles are scanned as part of the site-wide directory - ie. /

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

Component: base system » extension system

Just updating component.

ravi.shankar’s picture

StatusFileSize
new14.25 KB
new7.83 KB

Added reroll of patch #15 on Drupal 9.5.x., still needs work for #17.

smustgrave’s picture

@alexpott what is the correct order? Reading #17 seems - the installation profile directory; e.g., /core/profiles/standard needs to be removed but is that it?

smustgrave’s picture

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.