Problem/Motivation

When running the test suiet, specifically unit tests) as a user with write permissions to the Drupal root directory, the test suite creates a vfs: directory in the Drupal root.

The problem is caused by some code in Drupal\Component\PhpStorage\FileStorage that makes the assumption that when recursing a path at least one part of the path will exist because at some point it will reach the root of the filesystem.

Unfortunately the way this is implemented and the assumption itself can fail for streams like vfs used in the tests.

Proposed resolution

Refactor the Drupal\Component\PhpStorage\FileStorage::createDirectory method to make it more resilient to streams.

Remaining tasks

Patch
Tests
Reviews

User interface changes

n/a

API changes

n/a

Data model changes

n/a

Original report by neclimdul

Been meaning to take a look at this for... years? It happens, it should not happen, now there's an issue.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

neclimdul created an issue. See original summary.

neclimdul’s picture

neclimdul’s picture

Issue summary: View changes
Status: Active » Needs review

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

neclimdul’s picture

Still an issue... Fix still works...

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

jhedstrom’s picture

This makes sense to me. I think it looks good to go, just a few nits:

  1. +++ b/core/lib/Drupal/Component/PhpStorage/FileStorage.php
    @@ -149,8 +149,18 @@ protected function createDirectory($directory, $mode = 0777) {
    +    if ($scheme && "$scheme:" == $parent) {
    +      $parent_exists = TRUE; // We can't go deeper so it "exists"
    

    Nit: should this use ===?

  2. +++ b/core/lib/Drupal/Component/PhpStorage/FileStorage.php
    @@ -149,8 +149,18 @@ protected function createDirectory($directory, $mode = 0777) {
    +      $parent_exists = TRUE; // We can't go deeper so it "exists"
    

    Nit: This should be above the code line, and end in an '.' I think...

neclimdul’s picture

Goodness me, I don't really remember writing this anymore so I don't know why I chose == vs === but I can't see a reason not to. And yeah that comment is terribly placed. Again, no excuse :)

The comment isn't even that clear. I read through it a couple times and I think this version makes the logic a little clearer.

Thanks!

jhedstrom’s picture

I was trying to manually reproduce this issue. I ran vendor/bin/phpunit -c core/phpunit.xml core/tests/Drupal/Tests/Core/File/ but no vfs: directory is created in the root directory...

neclimdul’s picture

I don't think that test was actually creating the directory but something else using the FileStorage class maybe not even directly. I don't remember if I ended up getting to the bottom of that but probably not since I didn't put it in the summary. If you run the full unit testsuite you should see it but I'll have to dig in to find which exact tests are causing this.

jhedstrom’s picture

Status: Needs review » Reviewed & tested by the community

Ok, confirmed that running the entire test suite creates the vfs: directory. Also confirmed the tests in this patch fail (the directory is created) without the corresponding fix. Given that, I think this looks good to go.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

Given this is extremely low level code I'd like to make this change in the smallest possible performance impact as possible.

Can I suggest we change the patch to do:

    // If the parent directory doesn't exist, try to create it.
    $parent_exists = is_dir($parent = dirname($directory));
    if (!$parent_exists) {
      // Check if this is the root of some sort of stream. dirname() will strip
      // the trailing slashes from the uri leaving the scheme and a colon so we
      // use that for our comparison.
      $scheme = parse_url($directory, PHP_URL_SCHEME);
      if ($scheme && "$scheme:" === $parent) {
        // We can't go any higher so it "exists" and we'll let the stream sort it
        // out.
        $parent_exists = TRUE;
      }
      else {
        $parent_exists = $this->createDirectory($parent, $mode);
      }
    }

That way we only parse the URL when the directory doesn't exist which is the only time we actually need to.

And maybe we should consider doing something to not have to parse the directory all the times during a recursive call. Note this method is called a lot during a twig file cache rebuild.

  /**
   * Ensures the requested directory exists and has the right permissions.
   *
   * For compatibility with open_basedir, the requested directory is created
   * using a recursion logic that is based on the relative directory path/tree:
   * It works from the end of the path recursively back towards the root
   * directory, until an existing parent directory is found. From there, the
   * subdirectories are created.
   *
   * @param string $directory
   *   The directory path.
   * @param int $mode
   *   The mode, permissions, the directory should have.
   * @param string $scheme
   *   Used during recursive calls. Do not set.
   *
   * @return bool
   *   TRUE if the directory exists or has been created, FALSE otherwise.
   */
  protected function createDirectory($directory, $mode = 0777, $scheme = NULL) {
    // If the directory exists already, there's nothing to do.
    if (is_dir($directory)) {
      return TRUE;
    }

    // If the parent directory doesn't exist, try to create it.
    $parent_exists = is_dir($parent = dirname($directory));
    if (!$parent_exists) {
      if ($scheme === NULL) {
        $scheme = (string) parse_url($directory, PHP_URL_SCHEME);
      }
      // Check if this is the root of some sort of stream. dirname() will strip
      // the trailing slashes from the uri leaving the scheme and a colon so we
      // use that for our comparison.
      if ($scheme && "$scheme:" === $parent) {
        // We can't go any higher so it "exists" and we'll let the stream sort it
        // out.
        $parent_exists = TRUE;
      }
      else {
        $parent_exists = $this->createDirectory($parent, $mode, $scheme);
      }
    }

    // If parent exists, try to create the directory and ensure to set its
    // permissions, because mkdir() obeys the umask of the current process.
    if ($parent_exists) {
      // We hide warnings and ignore the return because there may have been a
      // race getting here and the directory could already exist.
      @mkdir($directory);
      // Only try to chmod() if the subdirectory could be created.
      if (is_dir($directory)) {
        // Avoid writing permissions if possible.
        if (fileperms($directory) !== $mode) {
          return chmod($directory, $mode);
        }
        return TRUE;
      }
      else {
        // Something failed and the directory doesn't exist.
        trigger_error('mkdir(): Permission Denied', E_USER_WARNING);
      }
    }
    return FALSE;
  }

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.