Provide better support for environment variables

Added a Pro about security. It links to https://12factor.net/config

Thanks for proposing this, @dawehner! Curious what people more knowledgeable than me wrt server security have to say :) I like the apparent simplicity!

How crazy / insecure would be be to totally pull settings.php from environment, which can also handle overrides already?

This topic reminded me of this article I read a while back that speaks to security of env vars. It appears that a few people in the article have varying opinions, but worth knowing some considerations around this topic.
http://searchsecurity.techtarget.com/blog/Security-Bytes/Environment-var...

This one has some good tips on securing env vars:
http://blog.honeybadger.io/securing-environment-variables/

For this article already mentioned in the issue description, I would like to see documentation around better ways to mange keys and other "secrets" than placing them in env vars.
http://movingfast.io/articles/environment-variables-considered-harmful/

I would also like to be able to specify a "DRUPAL_STATE" directory which is the only place that Drupal will be able to write things. Currently Drupal defaults to writing to sites, without an option to override that (except by using symlinks).

An environment variable would be the proper way to setup that config.

ps.
Drupal should also expect the directory to be empty when first started.

yeah, I think adding some support for dotenv would be really helpful and make the project appearing to play nice with industry-standards.

We've been using dotenv driven configs for a while, via symfony/dotenv which is added during the compose autoloader. Having environment variable defaults, allowing custom sites.php overrides and finally having per-multi-site dotenv that is applied across PHP and shell scripts ends up a bit tricky.
The setup we ended up with is testable from our drupal-project and works fine, but I want to improve how multiple dotenv files are combined in future:
https://github.com/drunomics/drupal-project/blob/4.x/web/sites/example.c...

I think some standard patterns that map environment variables to Drupal db credentials, Settings and Config would be definitely helpful.

Adding some tags after DrupalCon Seattle BoF on containerized Drupal in production. This would be a major help in not requiring everyone to come up with their own solution to 'how do I get Drupal's settings from the environment' and make Drupal a better player in the 12-factor app ecosystem.

Environment variables are also almost always prefixed and specific to hosters, e.g. plaform.sh puts stuff inside PLATFORM_VARIABLES, as base64 encoded json.

It's trivial to add a bunch of lines to extract it from there, we use d8config and d8settings prefixes to put stuff in settings/config overrides for example. But generic support wouldn't benefit us as we don't control the environment variables directly.

@Berdir - It would still be nice to have a common set, for anyone not using a hosting provider that is unable to provide the vars specific to Drupal. And it would help greatly for local environments as well, so all the Docker-based tools could just use the same set of env vars and a var like DRUPAL_ENV=local. (I think there's a separate issue to standardize support for Drupal 'environment' awareness though.)

There is a standard of sorts the drupal-project. Could be improved and more widely used, of course. See https://github.com/drupal-composer/drupal-project/blob/8.x/.env.example

We should probably not use .env at least until we move Drupal in to a separate document root. But then again, it is more secure to store environment variables on the server configuration rather than on the application level. For development and testing purposes its fine.

+1 On better support.

+1 On prefixes.

>But generic support wouldn't benefit us as we don't control the environment variables directly.

Imo, if you are using proper hosting, you should be able to control them - it's the industry standard for handling per-environment things, as they name "environment variables" say :D With Drupal propagating workflows with staging environments etc. this kind of seems a no-brainer to support.

Speaking of platform.sh, that allows setting custom environment variables as well.

We've been also using dotenv a lot in our projects and have a dotenv loader setup in https://github.com/drunomics/drupal-project to help dotenv file management and making sure .env is loaded across shell scripts and php the same.

Anyway, I highly agree that env variales should be standardized. We've started doing this across projects with a few essential ones here
https://github.com/drunomics/phapp-cli#available-variables
and more recently a few more here: https://github.com/drunomics/multisite-request-matcher/
We also have some helper to ease switching environment modes: https://github.com/drunomics/drush-phapp-env-mode

But in the end, what you want to cover with environment variables and what can stay fixed in some settings.php, differs per project. So having some optional variable support would be the way, e.g. something like

DRUPAL_ENV = {{ NAME }} (local,dev,prod, ..)
DRUPAL_ENV_MODE = production|development

which ideally are enforced to be set by code, and then optional ones for loading config like

DRUPAL_SETTINGS_{{ NAME }} = value
DRUPAL_CONFIG_{{ NAME }}___{{ KEY1 }}__{{ KEY2 }} = value
(only separators in env variables allowed is "_").

That pattern makes good sense to me.

oh, I almost forgot I also created https://www.drupal.org/project/services_env_parameter
Symfony has better, built-in support for environment variables in services.yml files, but would require some work in various places to make it work in Drupal, since Drupal's container builder is different to symfony's.

+1 to this issue in properly supporting .env.

We can even make this cleaner for true run-time changing of .env:

While currently the parameters in the parameter bag are included in the ContainerBuild definition [due to a Symfony build step], we could have Parameters that are of a NonIncludableVariable type.

Those variables would then be part of $build_definition['variables'] and can be changed at run-time, which means that they are loaded fresh together with the container to whatever value is in the environment.

It was one of the design goals of the Drupal's implementation of the Container [per alexpott' request at DrupalCon LA] to be able to specify parameters at run-time, but it has sadly not been used, yet.

Code - within likely a middleware - could then map from .env to container parameters.

Thanks, services_env_parameter looks interesting.

I came across this issue while looking for a twig function that could read environment variables for use in conditional statements eg. something like:

{% if getenv('VACANCIES_ACTIVE') is TRUE %}
  <h2>Apply Now!</h2>
    <p>
      We're currently looking for people to join our team, if you're interested
    </p>
    <a href="/jobs">apply now</a>
{% endif %}

and just in general, I like the idea of being able to enable maintenance mode via .env (eg. MAINTENANCE_MODE = TRUE).

It would be lovely to use a .env to set environment variables for testing. This would make it much simpler to swap between Drupal 10 and 11 for testing purposes.

I propose we add the Symfony dotenv library as a dependency and use it to load files from the root directory of the project. We could add

// loads .env, .env.local, and .env.$APP_ENV.local or .env.$APP_ENV
$dotenv->loadEnv(__DIR__.'/.env');

to core/tests/bootstrap.php so environment variables are available when testing and we could add it to \Drupal\Core\DrupalKernel::bootEnvironment() to ensure it is loaded when we boot.

This would allow consist environment management for Drupal core dev and projects built on Drupal.

What I have seen is that .env files are helpful for local development, and less so for staging and prod servers which have other means of writing env variables. Given that we have standardized on DDEV for local dev, which already support setting env variables from config or .env files, do we really need this? The OP was written in an age before Docker based local dev.

While ddev/docker does indeed support env variables natively, other hosting providers like Pantheon and the wild-west of bare metal and shared hosting do not have solutions in this space. Providing a core-supported option here might be the impetus for folks like Pantheon to switch from proprietary secrets text files to env variables.

I could easily see us taking the base added here and doing a lot of special things in follow-ups. Like injecting the DB credentials for the primary/default DB automatically. Or providing an env variable override process for settings and config. But the base logic/support would be needed first.

That sounds like a feature request for Pantheon (if their workarounds are not sufficient). Acquia and Platform support setting env variables, as do things like .htaccess and nginx config.

Like I said, DotEnv is really made for local development, so all this talk of web hosting demonstrates how people will be confused if this goes in. If people really want this feature, its a composer require away.

I could easily see us taking the base added here and doing a lot of special things in follow-ups. Like injecting the DB credentials for the primary/default DB automatically. Or providing an env variable override process for settings and config. But the base logic/support would be needed first.

I'm totally on board with settings.php honoring env variables. IMO thats the first part to do, not the last.