Config Environment module (original issue)

Thank you for opening this issue.

I think in a first step we need to define what the environment module is going to solve and what it is NOT going to solve.
I think it is important to keep the functionality simple to use and let more advanced use cases be handled in contrib.

We agreed that we want to support n environments but that at maximum only one environment can be active at the same time.
This is an important simplification of the problem space we need to make in order to be able to ship this feature. But this explicitly excludes support for using the environment as a hack for multi site setups.

So a site can be "in an environment" where one of the defined environments is active or a site can be in the NULL environment where no environment is active (for example an existing site before the environment module is turned on).

Here are the basic use cases I think we need to support:

• Switching a site into a given environment (this includes the NULL environment, ie no environment specific config)
• Exporting config while an environment is active (both zip download and with cli) but export is environment agnostic.
• Importing config while an environment is active, but not necessarily the same as the one from which the config was exported.
• Update code and retain configuration changes that have been done only in that environment when importing config.

I think the first three points are fairly easy to understand and relatively straight forward to implement. The importing and exporting can be done similar to how config_split works. Switching an environment is essentially the same as: exporting to a temporary config store, switching which env is active, importing from the temporary store (this is similar to how config_distro works).

However the update hooks changing configuration is tricky.
With config_split this is achieved by a specialised workflow which involves exporting the split which is only active on prod after the update hooks and before the config import or with config_ignore.

To illustrate why this is tricky let me explain the recommended workflow for updating code and config:

Introduce the code change:

1. getting the new code by writing it or by composer update
2. run database updates
3. export configuration
4. commit code and exported config
5. share/deploy

Other instances where the code is deployed:

1. getting code update (with git pull)
2. getting dependency code update with composer install
3. run database updates
4. import configuration

Of course there might be some more steps (ie you would do the composer install step as part of the build and deploy all the code as part of the artefact built etc).

The challenge is to retain the configuration changes that happen with the update hooks to configuration that only exists in the environment active where the code is deployed to. In other words in the production environment there might be a module that is enabled only there and it might have an update hook that is only ever run in that environment when the update hooks are executed since update hooks of non-installed modules are ignored. We don't want to revert the configuration that is changed in that specific environment but we also want to be able to deploy configuration updates for any environment.
We need to agree on a way to resolve this conundrum in a automated and predictable fashion that behaves in such a way as site builders would expect.

Adding a very rough work-in-progress patch.
It could serve as a proof of concept or a starting point for someone to flesh out the module.

Good to see initial work towards this goal. Suggestions:

• It would be best to explicitly address #2960961: Determine format for non-module configuration packages (splits) before digging into this work. We could gain a lot by modelling configuration packages in a way that opened the way to fuller support for extension-provided configuration packages.
• Introducing the concept of environments to core is probably useful, but this doesn't seem like the right place to do so. There is a large set of contrib modules that address different environment questions, including many that are not closely linked to configuration.
• Also, while environments might be the most common use case for configuration splits at present, there are others and tying the whole thing to environments may be unnecessarily restrictive.

Hi nedjo.
Thank you for your suggestions.
About #2960961: Determine format for non-module configuration packages (splits), I am not per-se against streamlining packages of configuration and the approach here (unsurprisingly) is the same as a split from config_split, except it is explicitly not in a separate folder but part of the sync folder the same way language overrides are. This is a consequence of wanting to support also a zip workflow. (In fact some of the code is straight from config_split)
I don't think moving forward with a solution that works for the use case needed to solve the stated problem impedes us finding a generic solution down the line.

About the set of contrib modules addressing environment needs: Yes there is a number of contrib modules including config_split that address some aspects of it. And I agree that there is more to environments than configuration. This issue is more or less about putting a simplified version of config_split into core though. So I am not sure what a better place would be. We can always expand it later.

Yes environments is the most common use case and that is why we focus on it. They are on purpose more restrictive than splits so that we can limit the scope and make assumptions that simplify the logic to deal with the different cases. We need to do this so that we can satisfy the higher standards for usability that apply to core. But this does not mean we limit the API to environments. In fact the API is being added in #2991683: Move configuration transformation API in \Drupal\Core\Config namespace and I added two patches here one with just the module and one including the config transformer patch from the other issue which is required to run it. This module we are proposing to add in this patch is meant to be the proof that the api works. You can think of this issue as "config_split in core" and #2991683: Move configuration transformation API in \Drupal\Core\Config namespace to be "config_filter in core".

For those of you who want to take the patch and flesh out the module and add a UI:
You can test the current patch by enabling the test module which will create a "dev" environment and then use drush to set the state config_environment.environment to "dev" and drush ev "\Drupal::service('config.exporter')->export();" to export the config, if you then set the environment state to something else the import will uninstall views_ui and the tour for it.

Edited by mpotter: Also enable the config_env_test_views test module to get the dev environment config entity.

And I agree that there is more to environments than configuration. This issue is more or less about putting a simplified version of config_split into core though. So I am not sure what a better place would be.

Ideally an environment entity would be decoupled from the configuration use case. Maybe call the new module Environment?

Adding a bunch of related core issues. #1219520: Support configuration environments and #2844681: Allow exported configuration to be environment-specific could maybe be closed as duplicates. #2879846: Provide better support for environment variables is probably also tangentially related.

So far in core any configuration can be provided by an extension. For example, an extension can provide language configuration collections. Would the same hold true for the new environment configuration collections? If so, how would that work?

I haven't read the issue in detail yet, but two thoughts for now:

• this reminds me a lot of #1537198: Add a Production/Development toggle, but I see @nedjo already linked to that! 👍
• this reminds me a lot of @e0ipso's https://www.drupal.org/project/environment_indicator module, I think he'll have thoughts about this issue :)

This module does not remove the need for Config Split ... on many sites. We are only trying to solve the very specific use-case of environment-specific config.

I see the attraction of a focused, achievable aim. But I still don't really see how this works. Do we want to encourage sites to run both a core module and a contrib module that do pretty much exactly the same thing? Or, alternately, do we want to be saying: if you need config packages only per environment, use the core module; otherwise, use the contrib one? Neither of these options seems ideal. When we bring modules into core, the idea seems to be to encourage people to migrate over the core version, which means, while the contrib original can be and always is stripped down, it still has to cover the basic use case, not a much narrower one that leaves the original use case hanging.

Looking at the draft code, beyond naming, I'm not spotting anything much that's linked to the concept or use case of an environment. If I wanted to use the module to create entities for any other purpose, I'd be fine. The only hard limit looks to be that, to be handled by the event subscriber, only a single config package can be active at a time. Which feels like an arbitrary restriction. Assuming this were to go into core, a contrib equivalent of Configuration Split might (a) add the ability to select multiple config packages rather than just one and (b) override the core event subscriber. Maybe that's okay, but if sounds messy.

Current environment ID will be stored in the State system. (implementation detail of config_environment service)

Storing the config ID in the state doesn't seem to fit with the guidelines for state data: "use State API to store transient information, that is okay to lose after a reset." Losing track of the current environment could cause data loss in the event of a fresh config import. #2879846: Provide better support for environment variables makes a good case for storing environment-specific data including the ID of the current environment in a per-instance file, such as a .env file read into the $_ENV array. Meantime, I suppose, we have settings.local.php.

I do not know how many sites run with several config splits at the same time. For all those that have just one this new core module will work out of the box. For those who use config split for multisite they will be able to use either just config split or a combination of config split and config_environment.

The main purpose at its conception was for config_split to be able to have development config that would be automatically be enabled (ie not be removed) when importing config but not be part of the sync directory that is used for a deployment. As such config_split can be enabled only on development machines and the fact that it is enabled can be split off. Of course it was clear that the code would be useful also in other cases and that using this module meant to trade off some of the safety/consistency of the configuration management for more flexibility. (I warned about this in the blog post introducing config split)
Splits and more generally config filters have a weight so that they can have a predictable order when several of them interact with the sync storage. But depending on how you configure your splits they it can lead to strange situations. For example if you have two active splits that blacklist the same module the configuration will be in the first split and the second split will not have the configuration available any more. So if you disable the first split and leave only the second split enabled you will have the module enabled but it will have its configuration missing. There are more of these gotchas that lead to some people complaining that it is too difficult to understand and I have all the sympathy for them. But unfortunately some of these idiosyncrasies will remain unless we significantly complicate the logic dealing with them. So limiting the scope to one environment is not an arbitrary choice but one that simplifies what we (at least initially) support in core.
This is also a reason why we call it "environment" and not split (which is meant to sound a bit scary) as a site is in only one environment at a time. Of course the transformer API which is the equivalent of config_filter will allow n amounts of event subscribers to interact with the config.

About saving which environment is active in state: We discussed this during one of the CMI 2.0 calls and indeed it would seem to make most sense to get this from a .env or the settings.php file and we have planned to add a warning if the value from the state does not match the value from the file system, and use it as a backup if the state is not set. But the sate system is much more adapted to what we need this setting to be because we want to support the workflow of downloading the database from production to another environment. And that would mean we need a system to switch the environment. On the same note we also need a way to enable or disable the environment as such. Changing the environment or turning it off or on means we install and uninstall modules. The current proposal with the transformer API works much the same way as filter and config_distro in that regard as enabling an environment means the active configuration is merged with the environment specific config and imported through the ConfigImporter. (Thus it can also fail to activate an environment if a module is not available for example.)
I am not sure in what circumstance one would "reset" the state. If you mean installing from configuration then yes that would currently install the config without the environment specific additions. But if we loose the state then the modules which are currently enabled (and the environment config) will get exported to the standard sync location, importing the configuration after loosing the state would just remove the environment specific configuration and uninstall the config.

I just had another idea instead of using state for the environment we could add a simple config object that holds the value and then unset it in the export transformation. that would have the advantage that if one exports or imports without the transformer api the system would not be confused and we could recover from it easily.

[edit] this patch was improperly tested...sorry! use #26

This patch fixes some omissions in #23, and completes the movement of all relevant core changes from #2991683: Add configuration transformer API into the config_environment module, so that all new functionality is self-contained within this module.

Changes since the last patch:
- added Add, Edit, Select, and Base form classes
- added stub for Delete form, submit does nothing
- added routes for Add, Edit, Delete, and Select
- added links to Add, Edit, Delete, and Select
- simplified SelectForm to just set the active environment for now

Next up:
- finishing up a comparison to active configuration on the Select form when an environment is chosen from the dropdown.

Does any one have a Drush patch? Mine is so-bad.

The constructor is definitely the wrong place for it cause the classes load all the time.

RE #28: look at #3036193: Add ExportStorage to allow config export in third party tools. This would make the drush patch very pretty and the patch for this issue could just decorate that service and apply the transformation there. Drush should not need to know about transformations in my opinion.

Removed ConfigTransformManager service, this is covered in issue #3016429
Removed StorageCopyTrait and associated test, this is covered in issue #3016429
Removed ConfigExporter service, this is covered in issue #3036193
Removed unused imports from ConfigTransformerSubscriber

To test, this patch requires:
#3036193 - Add ExportStorageFactory to allow config export in third party tools
#3016429 - Add a config storage copy utility trait and fix ConfigManager::createSnapshot

+++ b/core/modules/config_environment/config/schema/config_environment.schema.yml
@@ -0,0 +1,22 @@
+    modules:
+      type: sequence
+      label: 'Environment modules'
+      sequence:
+        type: integer
+        label: 'Weight'

+++ b/core/modules/config_environment/src/EventSubscriber/ConfigTransformerSubscriber.php
@@ -0,0 +1,256 @@
+    // Split the extensions.
+    if ($storage->exists('core.extension')) {
+      $data = $storage->read('core.extension');
+      $data['module'] = array_diff_key($data['module'], $modules);
+      $storage->write('core.extension', $data);
+    }

What about environment themes?

So i took the liberty to update the "API" part a bit as discussed during DrupalCon Seattle.
We don't want to expose the transformer as an API, because it is just an event that is dispatched.

General questions (Sorry if these are obvious, I'm new.)

• Is there a reason that the environment configuration is stored as a simple config type and isn't a configuration entity?

Follow-up to feedback from #31

• The label / machine name within the add form may also appear more standardized by using a machine name FAPI element.
• The machine name should probably have a unique check.

Re-focussed issue on UI only - see #3033427: [plan] Add support for environment-specific configuration