Option to have all php files outside of web root.

Drupal can't be used as a project's dependency until we figure out how to get Drupal out of the web root. We could do it with .htaccess rewrites.

There 2 major details for getting out Drupal of webroot I can think of right now (but there is probably a lot more):

• Get rid of DRUPAL_ROOT constant (or make it being relative) and include stuff by trusting the include_path (this one is optional)
• Copying all assets (CSS/JS/whatever) in webroot when enabling modules, or when requested (while enabling modules might be easier to achieve).

Once that done, it should be easy to make it work.

Yes it is, I hope JS guys (nod_ and others) will come to something with Assetic, it might help.

this can be a challenge to explain

I am currently testing D8 in Windows8 and it begins to fail after install. I am posting here because I think it is related. Let me know if I should create a new issue.

Basically once Drupal8 is installed all it does is give these Apache errors.

[Tue Feb 26 14:29:43 2013] [error] [client ::1] File does not exist: D:/wamp/www/drupal/user

However, if I change the URL to localhost:8888/drupal/index.php/user then everything works (somewhat) fine.

BTW, I am installing in a sub-directory (not using web root dir or v-host)
Apache Version : 2.2.22
PHP Version : 5.4.3
OS : Windows 8
WampServer 2.2

Edit : Created Issue.

I think that's probably more appropriate for a new issue.

Gotcha, new issue.

#2631362: Inject DRUPAL_ROOT into DrupalKernel would be super helpful for that.

It would be great at some point to get closer to Symfony and have the same variables being injected (aka %kernel.root_dir% etc...).

It would be great at some point to get closer to Symfony and have the same variables being injected (aka %kernel.root_dir% etc...).

Well, they are injected into the container indeed already, but there is stuff happening, before the container is there.

Closed a duplicate: #2765705: Move Drupal outside of the webroot

I suspect this sort of change would have to be a 8.2.x or even 9.x -level change.

The catch here has always been resource files.

With most code now made available through Composer/autoloading, that code can live anywhere. If you're using The Drupal Composer project, your vendor directory is already outside the web root. Much of Drupal core could easily move there simply by following through on #2352091: Create (and maintain) a subtree split of Drupal core, #2513388: Create (and maintain) a subtree split of each Drupal Component, and their ilk. and a little tweaking to the bootstrap process.

The catch is CSS/JS/image files, as those DO need to be web accessible. Symfony gets around that with a build step that copies or symlinks resource files from a bundle (Symfony module) to a web-public location. I don't know how well that process would fly with Drupal, though. That said, with Composer there is already a mandatory "build" step, so having it do one more thing in that process is less of a burden than it used to be.

Moving the sites directory (where DB settings and such are stored and generally little else these days), that would require either making it NOT optional or having some toggle that users have to edit in index.php themselves. The latter is not, necessarily, the worst fate, but is likely the only sustainable way to make it actually optional.

Yes, Symfony seems to use this code to handle moving the assets into place: https://github.com/symfony/symfony/blob/3.1/src/Symfony/Bundle/Framework...

I've opened an issue in the Drupal Composer Drupal Project to track this topic there as well: https://github.com/drupal-composer/drupal-project/issues/176

I also agree with Crell that the Composer-based build step offers a natural place to put the moving of static assets. Drupal Project even has a script that runs post-install/update specifically for creating required files:

• calling of script: https://github.com/drupal-composer/drupal-project/blob/8.x/composer.json...
• script: https://github.com/drupal-composer/drupal-project/blob/8.x/scripts/compo...

Static files could rsync or symlink from /core to /web/core in an additional script.

#1308152: Add stream wrappers to access extension files could be a big part of the puzzle, as it could allow to distinct between the folder on disc where the module lives exactly and the folder where the assets live. We need some abstraction then just drupal_get_path().

At least CSS/JS files should be doable, once the files are moved, because they are covered by our existing library extensions. I'm more worried about images and similar files.

I think we have to solve the ExtensionDiscovery part first #2701253: [meta] Make it possible to put modules into the vendor directory, modules and themes are currently not discoverable from the vendor folder. If me move core to vendor, we would loose all core modules.

@webflo
Do we really have to? Could we not also allow putting /core, /modules etc, basically everthing beside index.php outside of the webroot? In this case we don't have to support adding modules into the vendor directory.

@dawehner true, we don't have to put it in vendor. We can keep a separate folder but that makes the project root very crowded? core/, modules/, profiles/, themes/. What about sites?

Once/if we can put modules in an arbitrary location, there's little reason NOT to put them in vendor. That's where composer will put them by default if we don't tell it otherwise. Currently we tell it otherwise; we'd just need to stop doing that and all would be fine.

Assuming, of course, that everyone will be using composer. For users that are not using composer, the vendor dir is not really applicable. Also, we still need to support multi site.

Drupal extensions should *optionally* be able to live in the vendor dir.

I created a doc page how to run Drupal 8 with a separate docroot: https://www.drupal.org/node/2767907

As expected the most annoying thing is the rsync script to copy around asset files, but otherwise it was pretty easy.

Any Drupal 8 site that you maintain should be set up like this, much better site security where you avoid unexpected stuff in your docroot that attackers can access.

@klausi: That's excellent information, thank you!

Changing this to a "documentation" issue and requesting others review the information provided in the page klausi wrote.

I think there is still things we could do in Drupal core to help with this. @klausi's documentation is awesome, but it requires an rsync to extract the "asset"-like things from the Drupal modules and copy them into the docroot. It's using the file extensions to guess what is needed in the docroot - it doesn't actually know.

Ideally, we'd have a way for Drupal modules to describe which files need to be copied into the docroot, maybe in the MODULE.info.yml file or a new MODULE.assets.yml file or something? This would include things like CSS/JS used by asset libraries, as well as images, and the odd PHP file that is a separate entry point into Drupal like statistics.php.

Then we could have a script that actually knows what to copy into the docroot!

I think we should standardize on folders like css and js for our CSS and JavaScript, even a modules contains only a single file. We could start linking these two folders for every module and theme. Most contrib modules do it already, but it is still a mess in core.

find . -iname "*.js" | grep -v '/js/'

./book/book.js
./color/color.js
./color/preview.js
./comment/comment-entity-form.js
./content_translation/content_translation.admin.js
./field_ui/field_ui.js
./file/file.js
./filter/filter.admin.js
./filter/filter.filter_html.admin.js
./filter/filter.js
./language/language.admin.js
./locale/locale.admin.js
./locale/locale.bulk.js
./locale/locale.datepicker.js
./locale/tests/locale_test.js
./menu_ui/menu_ui.admin.js
./menu_ui/menu_ui.js
./node/content_types.js
./node/node.js
./node/node.preview.js
./path/path.js
./simpletest/simpletest.js
./statistics/statistics.js
./system/tests/modules/twig_theme_test/twig_theme_test.js
./taxonomy/taxonomy.js
./text/text.js
./user/user.js
./user/user.permissions.js
./views/tests/modules/views_test_data/views_cache.test.js

find . -iname "*.css" | grep -v '/css/'

./system/tests/modules/common_test/common_test.css
./system/tests/modules/common_test/common_test.print.css
./views/tests/modules/views_test_data/views_cache.test.css

We don't really care about JS and CSS, because most drupal sites will use aggregation anyway.

Important are the image files provided by modules and themes.

But still, a single location for any kind of asset would make a lot of sense. Convention over Configuration. No?

I agree it would be ideal to put assets in a subdir - css/ js/ images/ ? Should they be in an assets/ folder too so any other file types are handled? It's a simple and effective change.

What about creating symlinks on demand (as opposite to create them all on install) the same as image style is creating the derivatives. If a resource public://extension/mymodule/image/picture.png is missed, then Drupal takes the control according to .htaccess. And Drupal simply serves the file and creates the symlink modules/mymodule/image/picture.png > sites/default/files/extension/mymodule/image/picture.png.

I agree it would be ideal to put assets in a subdir - css/ js/ images/ ? Should they be in an assets/ folder too so any other file types are handled? It's a simple and effective change.

That would be awesome! I worry that that might be harder to retrofit on existing modules, though. Ie. moving all assets in core to an assets folder is a bigger, more disruptive change than adding some metadata somewhere. The same could be said for big contrib.

But, yeah, a separate directory would be ideal, and if we think we can swing it, let's do it!

Having Drupal create symlinks outside of the files directory at runtime would render it incompatible with Platform.sh; we use a read-only mount for code in production containers (which makes it more repeatable and more secure). Any symlinks or file copying or whatever need to happen before runtime, without database access, or we can't host Drupal. That would make everyone very sad. :-(

We also use nginx, not apache, as is the case for a number of hosting situations. That means .htaccess doesn't come into play.

The concept of an assets directory is worth considering.

Symfony bundles have a Resources/public/ directory the contents of which is copied or symlinked into the web root with an assets:install command.

I like the idea of modules having an assets directory regardless of the module's location (inside or outside the web root).

Creating symlinks on demand seems rather risky to me. The fact it's risky and that it wouldn't work for Platform.sh (b/c their system is designed for reliability/security) makes me feel like we shouldn't risk/consider that.

Thanks for the research in #33 about what is commonly done elsewhere! Let's definitely follow an existing pattern rather than making up our own. "Resources/public" ?

I think I like the directory name "public" more than "assets" because it sounds very, very clear that what you put in there will be publicly accessible to the web (and so you should be careful about what you put in there). And if Symfony bundles are already using it, I agree with @greggles, it's worth following an existing pattern.

Whether the default script copies or makes symlinks seems like an implementation detail - we could default to one and have an option for the other? Or, if we decide core's script copies and someone wants symlinks, it should be trivial to make a script for that in contrib. The hard part is deciding on the convention in the first place and adapting all modules/themes to it.

Probably could use an issue summary update..

To add to what @crell says in #32: regardless of whether it's at build time or runtime, I would expect symlinks themselves might fail on many platforms (Windows).

Copying files at build time (i.e. without bootstrap), based on a simple naming convention and/or a less simple config file for each module, would be stable. As greggles says, more research is needed about existing approaches.

For the record, we already have to rebuild some things in the files directory when we install/uninstall a module. It would not be unreasonable (IMO) to publish module assets (JS, CSS, images, etc) to an assets dir inside the files directory, and then optionally use mod_rewrite to make the URLs nicer. Personally, I think we should do that anyway so that we can just blacklist everything other than index.php and sites/ in our .htaccess (or Nginx/IIS/Caddy/whatever config). That's useful even if we're not moving all the php files out of the web root and might make a nice incremental step between what we have now and something more similar to Symfony's model.

Putting all asset files under files/ (or sites/default/files, or whatever) should be compatible with Platform.sh and other hosts like us with a read-only production file system. The caveat is that it would then need to happen on Deploy (after the code base is frozen, but you now have access to the DB and to the files mount). I'm pretty sure that's viable on our end.

If we can make that viable in some fashion that doesn't require mod_rewrite or equivalent, that would be better/more stable/more sustainable across the variety of hosts/web servers in the wild.

Random thought: What about having core register $filesdir/Resources/public (or whatever) much the same way that image styles work: A non-existing file gets mapped to a controller, which does a dumb file copy then returns a BinaryFileResponse() of the just-copied file. On a subsequent request, the file's already there, we're done.

Benefits:
* No symlink issues on Windows
* Respects read-only-codebase restrictions on 12-factor hosts (such as Platform.sh)
* No redirect logic needed
* No build step, so it's still compatible with users in no-build-step environments
* Does not preclude a build step to pre-copy those files if someone feels like writing it (and could even be done on module install)
* All assets are served from a single directory root, which makes file type restrictions in the web server easier to manage. (I.e., you know that only PHP will serve responses from /, and only raw files on disk or a mask from a PHP script in / will serve responses from /files.)
* Modules and themes can now live outside the docroot, even in vendor with everything else.

Drawbacks:
* As with image styles we'd likely need some sort of "cache clear" action when a module has been updated.
* This can *probably* be done with minimal BC breakage thanks to the libraries system, but not having touched that part of the codebase in a long time I'm not certain.
* Modules outside the docroot now becomes an interesting development problem for custom modules that you don't want a separate repository for. We would still need to support a non-vendor module location, even if it's not in the docroot.

If we're going to be messing around with abstracting asset file locations, we should probably take the time to look into Puli and see if it could help simplify the process at all.

@Crell, this I proposed in #30 but with symlinks. Of course we can copy instead symlink. I never said there that we should symlink/copy outside public://

Ah, that wasn't clear in the earlier comment. Copying at runtime to public:// should be fine (from a host perspective at least). Copying to docroot at runtime is a no-go. (Symfony copies files to the docroot, not to a writeable shares mount, which can only happen at build time on a read-only-prod host.)

@Crell, then each extension that wants to expose files to webserver will place the files in the public/ directory (eg. core/modules/views/public). On demand files will be copied to public://extension/views

core/modules/views/public > sites/default/files/extension/views

I'd just like to notice that some Drupal websites are still living on very small shared hosting plans that does not allow to change the docroot which is supposed to be the home dir of their FTP account. If we want to let these projects aside, we need to tell them somehow (and it might be considered as backward incompatible right now).

They can still run Drupal the old way, because that still works and will always work. This issue is just about making it possible with a separate docroot.

I like the idea of using public:// with a lazy copy (allowing preemptive copy as a build step).

One thing that would be useful from a security perspective would be a whitelist of at least file extensions that would be copied (ideal would be a .gitignore style syntax, but perhaps that could live in contrib).

I just noticed on a recent D8 project that image styles are broken if you move to an isolated docroot. Patch is in #2754273: Image styles fails with absolute path in $settings['file_public_path'].

They are also broken if you place them completely elsewhere, as the site serving them, not being a Drupal instance, will not by default catch the 404 and trigger the Drupal route.

The solution we used for this on a sports site has been to generate all derivatives on media (image) saves, by pushing the save event to a queue, handling it in a worker to avoid spending time on the site itself.

This link was in the Drupal newsletter today: https://github.com/drupal-composer/drupal-paranoia

A pragmatic implementation:
https://www.drupal.org/node/2767907

I think the best way is to configure webservers to process only the php files which are needed.

@C_Logemann that's definitely a good idea and one that Drupal site maintainers should adopt, but for defense in depth and effectiveness on a variety of platforms it seems important to also move all php files outside of the webroot.

We have composer project templates now. Why not start to adopt another composer project template to addition for existing drupal/recommended-project and drupal/legacy-project?

The current limitation is that drupal/core always has to be in docroot.

If drupal/core were location-agnostic, then it could be in /vendor and this issue would be solved for core.

Then we'd have to figure out how to allow extensions to live outside the docroot as well.

We have composer project templates now. Why not start to adopt another composer project template to addition for existing drupal/recommended-project and drupal/legacy-project?

Seems promising to me. I'll note that we have a scaffold composer plugin now - "The purpose of scaffolding files is to allow Drupal sites to be fully managed by Composer, and still allow individual asset files to be placed in arbitrary locations."

Followup of #1792310-84: Wrong DRUPAL_ROOT with non-standard code structure comment.

Composer scaffold already have a definition of the locations https://git.drupalcode.org/project/drupal/-/blob/9.0.x/composer/Plugin/S...

The following locations will describe current (legacy) project structure:

// Define any default locations.
$this->options['locations'] += [
  'project-root' => '.',
  'web-root' => '[project-root]',
  'drupal-root' => '[web-root]'
  'assets-root' => '[web-root]/sites/default/files',,
  'config-root' => '[web-root]/sites',
  'extentions-root' => '[web-root]'
];

The following locations will describe current (recommended) project structure:

// Define any default locations.
$this->options['locations'] += [
  'project-root' => '.',
  'web-root' => '[project-root]/web',
  'drupal-root' => '[web-root]'
  'assets-root' => '[web-root]/sites/default/files',,
  'config-root' => '[web-root]/sites',
  'extentions-root' => '[web-root]'
];

So in case if drupal will offer the project template with all requred stuff outside web-root folder the locations will look like that:

// Define any default locations.
$this->options['locations'] += [
  'project-root' => '.',
  'web-root' => '[project-root]/web',
  'drupal-root' => '[project-root]/vendor/drupal'
  'assets-root' => '[web-root]/files',
  'config-root' => '[project-root]/sites',
  'extentions-root' => '[project-root]'
];

So by introducing the composer core service which handle location configuration will replace usage of DRUPAL_ROOT constant.
Teoretically this will not break scaffolding functionality, so scaffold files can be located based on the configurated locations.

Also the extention assets files can be treated by #1308152: Add stream wrappers to access extension files