Running any tests which extended from BrowserTestBase getting permission denied

Try to run tests under the webserver user (differs per configuration), or root:
sudo -u [wwwrun|www-data|etc] /usr/local/bin/php ...

But I think all tests should be ran with core/scripts/run-tests.sh script.

I recommend reading core/tests/README.md which explain the problem around the webuser.

Strange, because my advice is based on core/scripts/run-tests.sh --help info. Thanks, will read core/tests/README.md

I still get this error when I try to run tests following the instructions here:
https://www.drupal.org/docs/8/phpunit/running-phpunit-tests

The problem I am seeing is that there is a PHP error on mkdir if parent directories don't exist, and because FileStorage->createDirectory() is calling mkdir() and not @mkdir(), instead of this condition being caught, it's causing the test to abort.

It looks like the intention was for the error condition to be caught and handled, because the status output of mkdir is being checked:

    // Otherwise, try to create the directory and ensure to set its permissions,
    // because mkdir() obeys the umask of the current process.
    if (is_dir($parent = dirname($directory))) {
      // If the parent directory exists, then the backwards recursion must end,
      // regardless of whether the subdirectory could be created.
      if ($status = mkdir($directory)) {
        // Only try to chmod() if the subdirectory could be created.
        $status = chmod($directory, $mode);
      }
      return $is_backwards_recursive ? TRUE : $status;
    }
    // If the parent directory and the requested directory does not exist and
    // could not be created above, walk the requested directory path back up
    // until an existing directory is hit, and from there, recursively create
    // the sub-directories. Only if that recursion succeeds, create the final,
    // originally requested subdirectory.
    return $this->createDirectory($parent, $mode, TRUE) && mkdir($directory) && chmod($directory, $mode);

However, unless it is called with @mkdir, the failure triggers a crash (this is when running phpunit, which has all PHP warnings turned into errors, and then PHPUnit itself aborts).

I actually think this is a major error. It also has a very simple fix.

Sorry, forgot the patch. This fixes it on my machine at least.

And by the way, I still got this error when running tests as root, which should not have had any mkdir problems. The problem is that the parent directory doesn't exist so it can't directly do a mkdir and it fails.

I'd like to be able to review this patch properly, but changing FileStorage feels super tricky. At least I would ask for a security review for the change.

Was getting similar issues on Gitlab CI - in our case we have a utility container (docker) that kicks off our functional tests and this patch fixed the issue. Nothing else seemed to work.

@rakesh.gectcr
Does this happen to you also on core tests?

https://github.com/Glavin001/atom-beautify/issues/1108#issuecomment-2720... explains what is going on ...

@rakesh.gectcr
So the solution in the github issue didn't worked out for you?

@rakesh.gectcr
How does the upload_tmp_dir PHP INI entry look like?

Looks like this patch is no longer applicable for 8.4.3, since some changes were made to FileStorage.php.

But the same error as before is still happening, as now there's an explicit trigger_error() call.

"Exception: User warning: mkdir(): Permission Denied Drupal\Component\PhpStorage\FileStorage->createDirectory()() (Line: 173)"

Issue where the change was made: https://www.drupal.org/node/2685957

Also wanted to note that I get these errors even when running as root, as well.

I should note that this error happens to me on some servers and not on others, so I'm having a problem replicating it locally. On the locations this error does happen:

- Running as "root" has no change. Running as the web user has not change.
- Running tests through run-tests.sh still produces the error.

It's very weird.

Yes, the errors I am getting have changed (see comment #24 here) since I did a git pull on the latest Drupal 8.5.x. And now I don't have a way to patch it.

In my case, I am running tests via vendor/bin/phpunit directly.

I am getting the error when the test framework is trying to create this directory:
sites/simpletest/27311886/files/php
I don't understand why, because I'm running as me, and the parent directory seems to be writeable by me while the test is running, although I guess it's possible that it has been chmodded to not be writeable at the moment the php directory is trying to be created.

As a note, the current code in this createDirectory() method starts out like this:

  // If the directory exists already, there's nothing to do.
  if (is_dir($directory)) {
    return TRUE;
  }

Then it does this:

    $parent_exists = is_dir($parent = dirname($directory));
    if (!$parent_exists) {
      $parent_exists = $this->createDirectory($parent, $mode);
    }

So... Here's a scenario to consider: if I'm trying to create sites/simpletest/27311886/files/php, and the parent directory exists already, but is not writeable, then it will fail. The function checks to see if the parent *exists* but doesn't modify permissions.

So... with that guess, I modified my local copy of FileStorage so it would modify permissions if the directory exists, and check parent is writeable. Then I got an error in chmod... so did a few more mods... also made the error messages show me which directory it couldn't create or modify. Here's what I ended up with:

    // If the directory exists already with the correct permissions, there's
    // nothing to do.
    if (is_dir($directory)) {
      if (fileperms($directory) !== $mode) {
        @chmod($directory, $mode);
        if (fileperms($directory) !== $mode) {
          trigger_error('chmod(): Problems modifying permissions on ' . $directory, E_USER_WARNING);
          return FALSE;
        }
      }
      return TRUE;
    }

    // If the parent directory doesn't exist, or isn't writeable, try to create
    // it.
    $parent_exists = is_dir($parent = dirname($directory));
    if ($parent_exists) {
      $parent_exists = is_writable($parent);
    }
    if (!$parent_exists) {
      $parent_exists = $this->createDirectory($parent, $mode);
    }

I'm now getting a message saying

chmod(): Problems modifying permissions on sites/simpletest/93732932/files

(which is the parent directory). Meaning the chmod() call failed on that directory. I really don't understand it, because the directory is owned by me, and while the test is in progress, I can see that it exists, and actually that it has the correct permissions. But who knows... I also tried running the tests as root, and had the same problem.

I didn't make this into a patch, because it didn't solve my problem. But this is broken for me, and others, making it impossible to run tests.

I guess I'll get the FileStorage class from before #2685957: Warning: mkdir(): File exists in Drupal\Component\PhpStorage\FileStorage->createDirectory, and apply my patch from #11 above so that I can continue to run tests... I am marking this as Critical though, because those of us with this problem cannot run tests at all, and that seems to me to be a crucial system that is not working for at least a subset of users.

It seems the new version of FileStorage is just too aggressive in triggering an error. I comment out the line "trigger_error('mkdir(): Permission Denied', E_USER_WARNING);" and my functional tests work again. Especially given what @jhodgdon has observed in #27.

I even set the correct parent directory to 777 with a umask of 000, with setgid. And *still*, I got this permission denied error.

To add some more meat to the mystery, I comment out the explicit trigger_error and run the FileStorageTest, and the *only* failure is the testCreateDirectoryFailWarning(), which is what you would expect. Every other test passes.

I was having the exact same problem, so I dug around and I do believe I've found the problem and a solution. I do not believe there is any bug here, rather just a config issue that looks like a bug at first glance.

My server is set up for multiple users with no sudo access, so I could not run phpunit tests using sudo or switch to the server's user as suggested in the README. All the server writable folders in my wwwroot are owned by the user and assigned to a group to which both the server and user belong. These folders also have the gid stickybit set. I followed the exact same steps as jazzdrive3 did in #30 and still my tests would fail.

Looking at the folder structure that was being created by the tests I noticed that the 'sites/simpletest/--number--' folder did not have the gid stickybit set. This was the source of the failures as all child folders could not be written as the group had been changed to user:user instead of user:server-user.

In Drupal\Core\File\Filesystem this is happening in the mkdir() method:

    // Do not check if the top-level directory already exists, as this condition
    // must cause this function to fail.
    if (!$this->mkdirCall($uri, $mode, FALSE, $context)) {
      return FALSE;
    }
    // Not necessary to use self::chmod() as there is no scheme.
    return chmod($uri, $mode);

I put a breakpoint between the call to '$this->mkdirCall(....' and 'return chmod(...' and found the gid stickybit was set after directory creation, however it was gone after the permissions were changed.

Looking at how the PHP chmod() function works, I discovered that the default which Drupal pulls from settings.php will eliminate any stickybits that are set:

 $settings['file_chmod_directory'] = 0775;

PHP's chmod() requires the octal value of 1000 to be set for the uid stickybit and 2000 to be set for the gid stickybit. Neither of these is present in the Drupal default.

I was able to get all my functional tests working by changing my default chmod to:

 $settings['file_chmod_directory'] = 02775;

Adding the 2000 octal value solved the problem where the chmod() call was dropping the gid stickybit.

As I mentioned above, I don't think there is a problem here, Drupal just drops all stickybits by default and that can be a problem for some setups.

I think we should make a small change to the README.md file to clear up any confusion on this issue.

@jgrunert is there anyway that we can test if this is going to be a problem and if instruct users what to do?

I unpatched my local FileStorage.php file (see comment #27), and (just to make sure), I ran one of my tests, and confirmed I am still getting the error described here (I'm on 8.6.x now, still the same problem).

But I tried @jgrunert's suggestion of putting

$settings['file_chmod_directory'] = 02775;

into my settings.php file (that line was commented out before, and said the default value was 0775, just as @jgrunert said). This did not resolve my problem.

@jhodgdon so the new approach in \Drupal\Component\PhpStorage\FileStorage::createDirectory() still causes problems for your setup? That code has been changed in 8.5.x / 8.6.x due to a race condition and now all the mkdir has is silenced. However I guess you are falling into to the:

        // Something failed and the directory doesn't exist.
        trigger_error('mkdir(): Permission Denied', E_USER_WARNING);

path. Or are you seeing something different?

Also just so we can workout what is going on @jhodgdon can you tell me what user is running your webserver, what are permissions on the /sites/simpletest directory, and what user is running the tests. And what command are you using to run the tests. Thanks!

I am running in the 8.6.x branch, in which I did a git pull/checkout about a week ago. The error I see is:

Exception: User warning: mkdir(): Permission Denied
Drupal\Component\PhpStorage\FileStorage->createDirectory()() (Line: 174)
(dir)/core/lib/Drupal/Core/Test/HttpClientMiddleware/TestHttpClientMiddleware.php:51
(dir)/vendor/guzzlehttp/promises/src/Promise.php:203
...

The webserver runs as user www-data, and I run tests as user jhodgdon.

Currently, sites/simpletest is owned by me, and ls -l shows drwxrwxrwx permissions, so anyone *should* be able to make a subdirectory in it.

The command I am using to run tests is, from the core directory, something like:

../vendor/bin/phpunit modules/help_topics/tests/src/Functional/HelpTopicTest.php

I think that about covers it... let me know if you need any more information.

See also #27, where I traced this deeper on this same (or at least very similar) copy of FileStorage.php. At that time at least, it was failing when trying to create a subdirectory
sites/simpletest/27311886/files/php
because for some reason, it had created the parent directories, but the immediate parent was not writeable. So it's able to create the first-level subdirectory, but the permissions are incorrect.

Is that because some of the startup process runs as me, and some as the Apache web server user?

@jhodgdon It looks like you're running a very similar setup to what I am. I have a question though, does your jhodgdon and your www-data user both belong to a common group? I also noticed that your ls -l output does not show the gid stickybit being set.

If you don't have the git stickybit set, basically what will happen is:

sites/simpletest - jhodgdon:www-data - 777 permissions
sites/simpletest/*number* - jhodgdon:jhodgdon - 775 permissions <--- at this point you've lost your group as PHP will always write folders as your user and group. The permissions are now 775, the folder is no longer world writable, so all subsequent directories will fail.

If you set the gid stickybit, the group will be forced to www-data (or whatever your common group is) and you won't lose the ability to write to that folder after that first directory is created.

I hope this helps!

@jhodgdon for me when I run tests in your situation I would do:
sudo -u www-data ../vendor/bin/phpunit modules/help_topics/tests/src/Functional/HelpTopicTest.php
which would avoid the issues around which process creates directories in sites/simpletest because all directories would be created by www-data

Hi @jgrunert, I don't know how to set the git stickybit. I tried putting this into my settings.php file but ... I am not sure that is used when I run tests anyway?

$settings['file_chmod_directory'] = 02775;

Anyway, that didn't solve the problem, my tests still failed.

And you are correct, I do not share a group between myself and www-data.

@alexpott -- I had tried running tests as www-data at one point in the past and ran into other problems... but that may have been with the run-tests.sh script, and using its browser output option, which I'm not using now since I could never get it to work (that's a whole other story), so... let's see. Yes, @alexpott, that does fix the problem. Interesting, I thought I had tried that before... So, that's great!

@jgrunert, if you run with sudo -u www-data (or whatever your web user is called), are you still having problems? Assuming your sites/simpletest directory is writeable by your web user that is...

@jhodgdon can we downgrade this from critical then because the current docs do outline the situation you faced? I do think they can be improved though since by far the simplest method is to use sudo -u <web-user> and that is not mentioned.

I believe @jgrunert's situation occurred in an environment where sudo was not possible so I definitely think it is worth improving the docs to cover that too.

Also I think we should open another issue to deal with setting up the browser output option as that is also super fiddly.

Fine with me.

New patch with proper line wrapping.

@jhodgdon If you want to set the gid stickybit, you would run chmod g+s /sites/simpletest once you do that, changing your default chmod to 02775 should fix your problem.

@alexpott is correct above, the environment that I'm using does not allow sudo access, that is why I had to figure this issue out. Using sudo -u www-data is the best way if you have access to sudo on your server.

I had tried it in the past with sudo -u www-data (my webserver user) and the tests still failed. So I don't think it's a solution to all of problems we're seeing.

This was inside a container where I had root access, which should have cut through all of this anyway.

I think the additional help text is nice.

Backported to 8.6.x as this is a docs improvement. Credited everyone who contributed to the discussion. Whilst I doubt this will fix all issues with running tests it does help some people so let's go forward and people can open new issues as they find them.

Committed and pushed 8500f748a9 to 8.7.x and 06df3f4357 to 8.6.x. Thanks!

I got this error in /drupal-test-traits/src/GoutteTrait.php:56 and it says $output_dir = getenv('BROWSERTEST_OUTPUT_DIRECTORY'); and in phpunit.xml I had <env name="BROWSERTEST_OUTPUT_DIRECTORY" value="/app/docroot/sites/simpletest/browser_output"/> and that was my issue. The error does need to be improved with a PR.

Created upstream issue to track > https://gitlab.com/weitzman/drupal-test-traits/issues/48

So, in my case even I have the same user, when I run phpunit and webserver. It still fails. Strangely, I still see that it creates other folders correctly, but it fails when it tries to create something like this:

User warning: mkdir(): Permission Denied sites/simpletest/56324300/files/php 

I tried to use proposed solution

$settings['file_chmod_directory'] = 02775;`

but it still doesn't work, after hours of debugging I've found that we have to use instead in default.settings.php

$settings['file_chmod_directory'] = 2775;`

I have no idea why it works. Maybe someone can explain this. I hope it will help someone

I have been getting the error

Exception: User notice: sites/simpletest/65500688/files/php
Drupal\Component\PhpStorage\FileStorage->createDirectory()() (Line: 134)

In order to trace down the source of this error I looked at the createDirectory() function in FileStorage.php. I could not get debug statements of various types to work. I searched and found #2795567: Use Symfony's VarDumper for easier test debugging with dump(). Then I made sure symphony/var-dumper was installed and applied the patch in #26. I put the following statement as the first line in createDirectory().

dump($directory);

This did not output the directory but the test ran successfully. I deleted this line and the test failed. Putting it back in the test ran successfully again. The output is in the browser_output sub-directory and not in a sub-directory with a random integer name. I am at loss since none of this is behaving as I would expect.

I spent an entire day just getting a short simpletest ported to PHPUnit. It should not take that long to port a test. I am inclined to suggest this be upgraded to critical.

This issue you've commented on is marked Closed/Fixed. If any of the people who have commented since it was fixed is having an error on the latest version of Drupal, you'll need to create a new issue.

Have the same issue
Guess, that it wasn't fixed

Thanks @LOBsTerr solved my problem #53

Having the same problem and managed to fix it.

In my case Drupal, PHPUnit and the rest inside docker container (docker4drupal if it's matter and for those who will google).

So, I connected to container (docker-compose exec php bash ) and run permission fix even if I saw that everything is fine with them:

find . -type d -exec chmod u=rwx,g=rx,o= {} \;
find . -type f -exec chmod u=rw,g=r,o= {} \;

I ran into this when running phpunit on Functional tests using BrowserTestBase in a custom module.

sudo -u www-data ./vendor/bin/phpunit --group web/modules/custom/

Created the simpletest folder and gave ownership to www-data user

mkdir -p web/sites/simpletest
sudo chown -R www-data:www-data web/sites/simpletest/

The tests only had Functional tests. Not sure what would happen if there are Unit tests as well. Maybe in that case we will need 775 on the folder and ownership to www-data:.

IMNSHO, I'd like to clarify and adjust @LOBsTerr answers on #53 (and #57). The correct value should be '02777'.
It seems you also need to allow "others" to write, that why the last '5' of '02775' should be turned into a '7'.

When you use '2775', you're not using an octal value but rather a decimal value (https://www.php.net/manual/en/function.chmod.php#31383). So what the system will understand will be 2775 decimal --> 05327 octal. In that form, "others" have read-write-execute access, which is what was needed, and the "sticky bit" (https://en.wikipedia.org/wiki/Sticky_bit) is set ('5' sets both sticky bit and temporary bit) which is also needed. So basically, by chance, 2775 works but does not setup the appropriate permissions for group and user.

The appropriate value should be '02777' which sets the "sticky bit" (2) and gives read-write-execute permissions to others (last "7") and everybody else (user and group).

That solved the issue for me, thanks to @LOBsTerr who pointed out (without understanding it? ;) ) that "others" needed write permission as well (and the "sticky bit"). 02775 did not the trick but 02777 did. Maybe the Readme should be corrected as well?

@guignonv Thanks for your answer. What a lucky coincidence. That makes me laugh :)

If you use docker4drupal, your need run test by root
Example:
Run test:
sudo -u root -E sudo -u www-data -E vendor/bin/phpunit --configuration /var/www/html/phpunit-core.xml --filter testRedirects web/modules/custom/only_admin/tests/src/Functional/RedirectTest.php

For debufging your need run next:
sudo -u root -E sudo -u www-data -E XDEBUG_SESSION=d4d sh -c './vendor/bin/phpunit --configuration /var/www/html/phpunit-core.xml --filter testRedirects web/modules/custom/only_admin/tests/src/Functional/RedirectTest.php'

If you use MAMP you should change the setting "Run servers as" from "Unix users" to your MacOS User to solve this problem.

Where to find this setting? In MAMP PRO 6 select in the SETTINGS sidebar the entry "Ports & User"

As described in #58 there is a problem when running Drupal in the Docker container. I use MacOS to run Kuberentes and Docker. Looks like no solution is working for me.

Lastly, I'm using a solution with bash script that execute code in PHP container under sudo user and everything is fine.

The solution below is tested and done for Docker4Drupal containers, but I think it will be easy to adjust for your needs on other containers.

#!/bin/bash
# This is special wrapper for PHP executable inside wodby/php container.
# It's intended to be used instead of 'php' executable in PHP storm to be able
# run Functional tests, that require that user.
#
# @see https://i.imgur.com/pYZoXcx.png
#
# If you have error 'Directory "/opt/phpstorm-coverage" could not be created',
# then you should help it to be created:
# 1. $ sudo mkdir -p /opt/phpstorm-coverage
# 2. $ sudo chmod 777 /opt/phpstorm-coverage
#
# This script should be executable to work:
# $ chmod +x d4d-php.sh (or your script name)
sudo -u root -E sudo -u www-data -E /usr/local/bin/php "$@"

You also can find screenshot (important part is PHP executable setting) how to configure PHPStorm to use that script and then tests are perfectly ran in it.

Also, I have added Composer scripts to simplify usage from CLI.

    "scripts": {
        "d4d-phpunit": "scripts/d4d-php.sh vendor/bin/phpunit",
        "d4d-phpunit-coverage": "scripts/d4d-php.sh vendor/bin/phpunit --coverage-text"
    }

I use that approach even in GitHub Actions and it works perfectly.

So I think I figure out what is going on.

1. I run PHPUnit as root in my container.
2. It creates `sites/simpletest/78851319` directory in the set up.
3. The permissions of this directory are messed up:
```
drwxrwxr-x 2 root root 4096 Aug 4 20:34 78851319
```
4. When simpletest runs it makes a browser request to a Drupal.
5. Drupal is running as www-data.
6. Drupal tries to create a twig directory `sites/simpletest/91484958/files/php/twig`
7. This is where it fails because `www-data` user can't create anything inside `sites/simpletest/91484958`

As a solution I was trying to set a sticky bit to `sites/simpletest` directory so permissions of it looks like following, however this solution did not work.

```
drwxrwxrwt 2 root root 4096 Aug 4 20:44 simpletest

```

Also I figured out that `sites/simpletest/91484958/settings.php` is based on `core/assets/scaffold/files/default.settings.php` and not on `sites/default/default.settings.php`, however adding `$settings['file_chmod_directory'] = 02775;` into it did not help, because this file (`core/assets/scaffold/files/default.settings.php`) is not used as a source for copying. Looks like Drupal is getting this file from somewhere else. I am not sure where should I place `$settings['file_chmod_directory'] = 02775;`.

UPDATE 1

Got a little progress. Added following lines into "settings.php":

```
$settings['file_chmod_directory'] = 02777;
$settings['file_chmod_file'] = 0666;
```

Now `sites/simpletest/74758038` is created with permissions I need:

```
drwxrwsrwx 5 root root 4096 Aug 4 21:08 74758038

```

However `sites/simpletest/74758038/files` permissions are not right:

```
drwxrwxr-x 3 root root 4096 Aug 4 21:08 files

```

UPDATE 2

So running as `www-data` user seems to solve the problem (sudo -u www-data /var/www/vendor/phpunit/phpunit/phpunit). However in this case you need to install sudo into container. Alternatively container can be run as www-data user.

same #53 fix my problème

Finally, found reliable solution for Docker4Drupal environment.

Add new environment variables for php container in docker-compose.yml:

  php:
    image: wodby/php:$PHP_TAG
    container_name: "${PROJECT_NAME}_php"
    environment:
      …
      PHP_FPM_USER: wodby
      PHP_FPM_GROUP: wodby

Then rebuild container by docker-compose up -d --no-deps --build php and that's it. All tests, including Functional, will be run without needing using sudo or anything else. They just run as expected, even from PHPStorm.

#68 works.
Thank you @Niklan!

Thanks @Niklan!

I was struggling with this issue but your solution works like a charm.

One possible solution is setting directory permissions manually.

The attached patch assumes you have the following configuration in settings.php file.

if ($_SERVER['DRUPAL_FUNCTIONAL_TEST'] ?? FALSE) {
  $settings['file_chmod_directory'] = 0777;
}

Note, you will still need to set permissions for sites/simpletest/browser_output directory. Also ensure access to SQLite database.

Actually, it's better to follow #53 but use settings.testing.php file instead.

$settings['file_chmod_directory'] = 02777;

For me, it was the permission hardening that was setting the simpletest directory to 0555. Simply adding this to default.settings.php worked:

```
$settings['skip_permissions_hardening'] = TRUE;
```