FileSystem::mkdir should handle open_basedir correctly

Ran into the same problem, and was able to track it down to \Drupal\Core\File\FileSystem::mkdir(). This method iterates through all the components of the file_private_path and checks if the component exists. Since in our case the first component is .., file_exists() will return FALSE, thus this method tries to create it, which will obviously fail.

By the way, the same bug can be triggered by trying to download an image derivative whose directory does not exist yet. In this case, the error message is different: "Failed to create style directory: %directory" (from \Drupal\image\Entity\ImageStyle::createDerivative()).

A patch is coming, but I don't know how to write a test case for this, as the existence of this bug highly depends on hard-to-change PHP settings.

The bug is still present on 8.2.5.

Maybe relate to thus issue: https://www.drupal.org/node/2878778

IMO, the right fix is to call createDirectory() being in core/lib/Drupal/Component/PhpStorage/FileStorage.php or to duplicate its behavior, as it already handles open_basedir correctly.

@LpSolit: could you submit this idea as a patch?

Here is what I had in mind.

Fix the error message in core/modules/migrate/tests/src/Kernel/process/FileCopyTest.php.

Updating the bug summary to better reflect what the problem is. With my patch, there is no need to use relative paths. Absolute paths just work fine.

Facing problem. Buzz for the fix.
For the #7, for the directory create, you should add DIRECTORY_SEPARATOR like this:

if ($component == '..') {
          $recursive_path .= DIRECTORY_SEPARATOR;
          continue;
        }

I just ran into the same problem and can confirm this issue.

I experienced this issue on a Plesk server where paths are like:

/var/www/vhosts/domain.com/httpdocs (DocRoot)
/var/www/vhosts/domain.com/private (the private folder I created not accessible over the web, ownership domain-user:psacln, permissions 775)

With the absolute path Drupal was able to create the .htaccess correctly, but any uploads failed because it was not able to create subdirectories.

$settings['file_private_path'] = '/var/www/vhosts/domain.com/private';

Using #7 (thanks @Boobaa and @tobias-hd) and adjusting it with #17 (thanks @trungonly) allowed the private directory to work correctly using:

$settings['file_private_path'] = '../private';

For anyone on a shared host or with setups like the above this is a good solution that can be easily implemented. I spent several hours trying to figure out why private files did not work at the server level.

Many thanks for the suggestion. Struggled with this for hours ...
A relative path is the answer!

Thanks #21 for saving me hours of debug... Much appreciated !

Thanks. I came up against this issue when I added a file upload field to a webform. A relative path for the private file directory made it work. I didn't apply any patches.

Thanks all, I was wrapping my head about this issue as well on a Plesk-based Drupal installation. Providing the private filesystem folder with a relative path solved the issue for now without any patches applied.

This worked for me too. I had to apply the patch AND set a relative path.

I was using ISPConfig on a Debian server here.

Also on ispconfig - came across this patch after banging my head against the wall for hours. Thanks!

Thanks to all!!
After a lot of time debugging the issue on a Plesk-based Drupal installation, setting the private dir to relative solved the issue (without any patches applied).

might this patch be making its way into core soon?

I for one still need this patch.

Let's try that again.

#32 applied cleanly and tests passed. Moving to RTBC.

+++ b/core/lib/Drupal/Core/File/FileSystem.php
@@ -187,7 +187,14 @@ public function mkdir($uri, $mode = NULL, $recursive = FALSE, $context = NULL) {
+    ¶

Unexpected leading spaces and should file against 8.9.x or other branches, not 8.8.x.

Failures seem unrelated and the re-testing passed, and #36 is just a tiny trivial change, so setting back to RTBC as #34.

Corrected sentence from 'If the directory exists already, there's nothing to do.' to 'If the directory already exists, there's nothing to do.'

Thanks, @Neslee Canil Pinto!

Re #38 It can be either in the mid position or the end position actually. But already in the mid position seems more formal than in the end position.

Hello @jungle.
Your point was right, in mid position seems formal.

I have reroll patch #38
Kindly review it.

Rerolled patch for 9.3.x-dev

Changed path of #46 for it to apply and added interdiff.

The only way that test is passing is increasing the sleep time...

Added note about open_basedir in docs : https://www.drupal.org/docs/8/core/modules/file/overview#s-managing-file...

Attempt to make it pass the testNonWritableDestination

As it is passing right now, please consider review the patch.

I'm using this patch for years without any problem. I needed it to make the filebrowser module work with open_basedir. Not sure if this is enough to RTBC this patch.

aleix's patch #54 works fine with the filebrowser module and is 99% similar to what I wrote in comment #15, which I use for 5 years now. Not sure if I'm allowed to RTBC this patch.

This issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue [2700+ issues] to around 400 (1 month or less), following Review a patch or merge request as a guide.

The issue summary should be updated to clearly show what the proposed solution is, remaining tasks, etc

-        usleep((int) (($time_to_start - microtime(TRUE)) * 1000000));
+        usleep((int) (($time_to_start - microtime(TRUE)) * 5000000));

Not sure if this is in scope?

But as a bug this will need a test case showing the issue.

Thanks for the patch, without it, it is impossible to handle securing private files outside of the webroot with the open_basedir directive. After applying it, wet can set private and tmp using relative directories, such as:

$settings['file_private_path'] = '../../private';
$settings['file_temp_path'] = '../../tmp';

After some time getting to the bottom of #3411785: Create subtheme doesn't work with open_basedir restriction, I would hope that we could get this sorted out. @smustgrave: is it possible to set up open_basedir in the testing environment? If not, I'm not clear how a test could be created for this issue.

I use shared hosting for a site, but I am able to use absolute paths, even when I set the file private path. Maybe the issue summary should explain when there are issues with using absolute paths.