Installation fails with files directory on glusterfs: "Warning: mkdir(): File exists"

Was just informed by Fabianx this might be resolved by the patch in #2497243-188: Replace Symfony container with a Drupal one, stored in cache.

@cilefen - I've also tried the following scenarios, none of which worked:

• Running UI installation through Nginx balancer with ip_hash balancing strategy (which pins all requests for one IP to the same backend).
• Running UI installation directly on one of the backends (bypassing balancer entirely).
• Drush si on the backend directly (this was the initial failure that sounded the klaxons).
• Running UI installation through Nginx balancer with default/round-robin balancing strategy.

None of those worked, but all of them worked fine once I switched to beta11. I'm retesting with Fabianx's patch in the related issue.

Could be... I've been reading through the following:

• http://irclog.perlgeek.de/gluster/2013-11-27 (someone else with the exact same issue
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=697800 (a linked bug in Debian, relating to the issue)
• https://bugs.php.net/bug.php?id=48099 (a linked bug in PHP, relating to the issue)

It looks like it could be an issue with the version of Gluster available for Debian Jessie, so I'm going to switch back to NFS and do a little more testing.

Of course, this brings up another fun topic: why the heck are we doing so many stats and mtime lookups in Drupal 8? Ugh. A prescient quote from efflugentsia 5 years ago in #678292-22: Add a development mode to allow changes to be picked up even when CSS/JS aggregation is enabled:

...with Linux file system caching, mtime is probably cheap enough to not worry about. But, we shouldn't assume a standard Linux file system. There's NFS, GlusterFS, and countless other server configurations. On some of them, file stats are noticeable. That's why APC provides a setting to turn them off, and so should we.

But I digress. Still digging here.

Full trace of the error that's wasting away my life right now:

Warning: mkdir(): File exists in Drupal\Component\PhpStorage\FileStorage->createDirectory() (line 171 of core/lib/Drupal/Component/PhpStorage/FileStorage.php).
Drupal\Component\PhpStorage\FileStorage->createDirectory('/var/www/drupal/sites/default/files', 511, 1)
Drupal\Component\PhpStorage\FileStorage->createDirectory('/var/www/drupal/sites/default/files/php', 511, 1)
Drupal\Component\PhpStorage\FileStorage->createDirectory('/var/www/drupal/sites/default/files/php/service_container', 511)
Drupal\Component\PhpStorage\FileStorage->ensureDirectory('/var/www/drupal/sites/default/files/php/service_container')
Drupal\Component\PhpStorage\MTimeProtectedFastFileStorage->save('service_container_install_17922066.php', '<?php
namespace Drupal\Core\DependencyInjection\Container\install;

use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\DependencyInjection\Container;
use Symfony\Component\DependencyInjection\Exception\InactiveScopeException;
use Symfony\Component\DependencyInjection\Exception\InvalidArgumentException;
use Symfony\Component\DependencyInjection\Exception\LogicException;
use Symfony\Component\DependencyInjection\Exception\RuntimeException;
use Symfony\Component\DependencyInjection\ParameterBag\FrozenParameterBag;

I have the files directory symlinked to /mnt/gluster/files, which is a mount of a GlusterFS share. I've now tried GlusterFS 3.2.x and 3.5.x. Also hitting the issue with NFS shares mounted and symlinked.

And through the UI / Drush, I'm sometimes also getting the following during requirements phase:

Writable (public download method)
The directory sites/default/files does not exist. An automated attempt to create this directory failed, possibly due to a permissions problem. To proceed with the installation, either create the directory and modify its permissions manually or ensure that the installer has the permissions to create it automatically. For more information, see INSTALL.txt or the online handbook.

I have (as stated above) sites/default/files set as a symlink. No clue why it's not working here; I can read/write to files in there with no issue, and installing different Drupal 8 tags/commits (always with a clean slate and re-symlinked directory) gives different results :P

So the rabbit hole gets a little deeper... Seems like PHP is possibly the culprit.

I created a quick test file in the docroot:

var_dump(is_dir('sites/default/files'));
var_dump(is_file('robots.txt'));

And running it (from either command line or through webserver/Nginx) returns:

bool(false)
bool(true)

From the same directory as that file, I can see the directory is clearly a symlink:

# ls -lah sites/default/files
lrwxrwxrwx 1 www-data www-data 18 Jul 28 04:35 sites/default/files -> /mnt/gluster/files

So... now I'm trying to see what kind of settings in PHP might be causing the symlink to not appear as a directory.

Also note that:

• is_link() returns true
• is_writable() returns true

Maybe we are a bit too brittle with our installer requirements for the files directory?

Attached patch allows Drupal to install successfully, and when used in tandem with the patches in the other issues mentioned previously, Drupal 8 HEAD finally runs on the #Dramble again!

So... I'm surely not the only person who's ever run Drupal with a symlinked files folder; is there any better way to fix this? Am I missing something obvious in the PHP config?

Note that after Drupal installation is complete, I'm still getting Twig-related errors like the following:

Warning: filemtime(): stat failed for /var/www/drupal/sites/default/files/php/twig/1#f7#ef#f11532315b88bd04b9c3a481a608fafdfd0f381904ae0470d9457c6f1a62 in Drupal\Component\PhpStorage\MTimeProtectedFastFileStorage->getUncachedMTime() (line 192 of /var/www/drupal/core/lib/Drupal/Component/PhpStorage/MTimeProtectedFastFileStorage.php).

Warning: mkdir(): File exists in Drupal\Component\PhpStorage\FileStorage->createDirectory() (line 182 of /var/www/drupal/core/lib/Drupal/Component/PhpStorage/FileStorage.php).

I've also set all permissions on the gluster mount directory (including the files directory within) and the symlink to 777 with www-data as owner/group. Still no dice.

I'm going to close out this particular issue, because I've finally thrown in the towel on getting Gluster working with shared files on the Raspberry Pi. Too many little problems, and I finally switched to NFS: https://github.com/geerlingguy/raspberry-pi-dramble/issues/55

Ah well... at least I tried.