Problem/Motivation

Everyone is having problems with modules being exported where they don't want it to be. The canonical example is the devel module.

Proposed resolution

Add a key to a module's info.yml file to allow them to opt out of config deployments.

Remaining tasks

User interface changes

None

API changes

TBD

Data model changes

None

Comments

alexpott created an issue. See original summary.

nicrodgers’s picture

Huge +1 for this! Instead of making it an option in each module's yml, could we have a new config setting for the site called "ignored modules" or something similar?

swentel’s picture

Just wanted to briefly document how https://www.drupal.org/project/config_split works: The config split config entity defines what is blacklisted, allowing you to blacklist modules and/or config objects. There's also graylisting, but my head isn't fully wrapped around a use case I can describe nicely (fabian can comment on that)

The canonical example, being devel, already has an interesting use case: it comes with system.menu.devel which, in case you blacklist devel, the menu config file will not be removed since there's no dependency. While it's not a major problem, config split allows you individually select that as well so it's removed on the environment.

By using config entities, it allows you to have multiple 'splits' and go in any direction you want: developer a to developer b, dev -> production, production to dev ...

I know there's no patch yet, but using a flag in the info file doesn't sound like the best solution for this feature and would only solve one direction, and at the same time potentially introduce more WTF moments. And hook_system_info_alter() is kind of annoying to control this flag.

vdanielpop’s picture

Issue summary: View changes
mallezie’s picture

Another use case (but the reverse aproach) would for example be contact forms which can be edited on production and not reverted on config import.
Perhaps a more extreme one in this bucket could be config translations.

alarcombe’s picture

I created this module recently to scratch a related itch. https://www.drupal.org/project/environment_config

In the case where you have multiple environments (say dev, qa, stage, prod1, prod2) with mostly the same configuration (eg same content types, permission etc) but you wish to override some settings on a per-environment basis. Commonly this is stuff like smtp config (dev, qa might use localhost, stage, prod1, prod2 might use third party services), api keys (dev, qa, stage, prod1, prod2 might all need a diff key per host), syslog (dev, qa, stage might just identify for local syslog, prod1, prod2 might use log aggregation services and need to identify which host messages are coming from.) You get the idea.

This module essentially enables a directory of config overrides - the directory is specified by an environment variable and so can be configured from outside of Drupal (eg in ansible, chef etc).

Conceptually it's like Drupal CMI is configuring your domain specific Drupal application (eg by defining the content types and permissions it exposes to the world), and this module sits atop that in order to configure your domain specific Drupal application for the particular environment that it is running on.

This approach is more for building on top of an existing configuration, but could potentially be used to remove configuration?

scott_euser’s picture

You can also use drush cexy and drush cimy to exclude some config from import and export and you can control that via drush/config-ignore.yml.

cilefen’s picture

I agree with #2, #3 in that there are two situations I suppose we would be looking for in core:

  1. Devel modules who would never ever export configuration could specify that. The module author would have to be certain about it in a world where this is laid down in the module info file.
  2. Site-specific reasons for not wanting to export the configuration of any given module. There are a few implementations of this noted above.

The first situation scares me just a little, for the WTF reason mentioned in comment #3. Perhaps the info file setting could be the default stance of a given module about its config exportability but that could be overridable by import/export alteration approaches like https://www.drupal.org/project/config_split and drush cexy.

alexpott’s picture

@mallezie I think we should limit the scope of this to allowing modules to opt out from the config-export / config-import process and no reworking the more advanced functionality of https://www.drupal.org/project/config_split. The issue for devel affects everyone. The issue for config translations / contact forms etc depend on a site's administration workflow and that can be unique to each site :).

mallezie’s picture

@alexpott sounds fair. Just posting here to keep in mind to allow perhaps something in future iterations.

bircher’s picture

I am glad we are having a discussion about this topic on the core issue queue. It is a problem space I have thought about for quite a while so I will chime in a bit.

Firstly adding a "development" flag to the modules info file as such is not a bad idea. It could for example be used to warn users not to run them on their live site. But using that for managing configuration has a couple of important drawbacks. As acknowledged in #3 and #8, it becomes the module maintainers decision and not the site owner and this seems to be a departure from "the site owns the configuration" mantra that makes configuration deployment so robust.

A simple flag alone is not going to cut it even for the canonical use case of the beloved Devel module. Devel comes with a menu that it sets up and properly removes when uninstalled, but the menu does not depend on devel and can not be detected at the moment.

Another consideration to take into account is that we would then need to define (I guess in settings.php) which "environment" a site is in to detect whether the development modules should be enabled or not. This would be akin to `require` and `require-dev` with composer.

It is then also important to be able to share the development configuration between developers (or yourself if you re-install the site or restore a backup of the live site). So the development configuration still also has to be exported.

For all of this to work we need a concept of "development configuration" (not just modules) and for that to work we need a proper API that can do exporting (and importing) with the development configuration enabled or not.

So rather than a flag in the info file I think we would need another key in the `core.extensions` configuration (like in composer.json) and a flag in the configuration to mark it as "development" if it is not depending already on a module in the "dev" section.

Alternatively we can choose an approach as in config_split and change/override/filter configuration when importing and exporting, between the current "API" and the files. It allows for more flexibility with not only a binary system but splits as configuration entities and a set of ignored-when-overwritten configuration (the behaviour of --skip-modules in drush) Btw the import form the UI works with config_split when switching out the `config.storage.sync` in the settings.yml and I am considering just doing that with a ConfigSplitServiceProvider. This approach is very similar to what drush does with `--skip-modules` but takes it all the way for all configuration.

alexpott’s picture

Well the fact that the devel menu block does not depend on devel is probably a bug... actually it does :)

id: devel
label: 'Development'
description: 'Links related to Devel module.'
langcode: en
locked: true
dependencies:
  enforced:
    module:
      - devel

I was proposing that we'd remove all config that depended on such a module. So this would be excluded too.

@bircher thanks for your thoughtful comment The argument about sites owning configuration is a good one.It is tricky though because nobody wants to deploy a devel so everyone is reaching for solutions. I like the idea of a dev section of core.extension. I think the point about not being able to import and share dev settings is also a good one. I don't think this issue should solve the dev/prod toggle argument but I really wish that we hadn't spent ages arguing over an extensible system when really just implementing a binary system would have given as so much.

I think there might be a nice way around the challenges @bircher talks about. Instead of affecting config-export we just concern ourselves with configuration import. If we detect configuration that depends on a dev module we ask the user - "do you want to import it?". That way we can avoid not exporting the full site and we still put the site owner in charge.

alexpott’s picture

We can even only ask this question when the module is going to be installed which is kinda neat.

bircher’s picture

Oh yes I was not using the latest version of devel.
So that is great then, because if we have a way to make some modules "dev" then people can just make their development configuration force-depend on one of their "dev" modules.

And yes we can export as usual if we just keep the "dev" config somehow inactive (I would still love to have a better API, but maybe for another issue). And we can sort out the "dev" modules and their dependencies on import.
I am not sure what you mean with

only ask this question when the module is going to be installed

.
But I think "installing" gives the wrong impression. And I am not sure it is correct.
Rather, when importing config when the site is in the "no-dev" state all the "dev" modules should be in the "will be uninstalled" or "keep uninstalled" camp and all the configuration that depends on one of these modules should be in the "will be removed" for active config and "ignored and not added" for config only in the sync folder.
While when the site is in the "dev" state, the "dev" module list and the current list should just be merged and all config imported as we do now.
That way when there is configuration that depends on a module that will not be installed we can still have the error.

Actually we probably still have to import the "dev" configuration and just have it inactive and unprocessed. Unless we decide that the following scenario is unsupported: Exporting from production (changes by clients) and merging back to development.
Since if we export from a "no-dev" state the "dev" configuration should still be there.

Of course we can also take a conscious decision not to support that use-case. config_split would continue to work the same way with the solution I am proposing here and it would cater to that use case still so in these situations config_split would still work as it does now.

geerlingguy’s picture

I've tried a few different methods of managing environment-specific config, and config_split seems to hit the right usability vs. extra overhead balance the best so far. It's simple and lightweight, but allows me to mark (and continue to use) certain configuration only in certain environments. I'm not sure if we need to necessarily make it easy for some modules to opt out of config export entirely if we can make it easy for people using CMI to exclude certain configuration in general. I think it would be a perfect addition to core CMI to just add in config split once it's gotten a little more stable, as part of the core CMI UI...

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

gambry’s picture

At DrupalCamp London I presented a configuration "segmentation" model, after reviewing what popular contrib modules do and understanding why for a standard developer/site-builder the standard synchronisation process is not enough.

Here the summary of the talk, check the slides for additional info of each segment.

The argument about sites owning configuration is a good one.

If we detect configuration that depends on a dev module we ask the user - "do you want to import it?". That way we can avoid not exporting the full site and we still put the site owner in charge.

That's the core of everything.

My investigation goes further than just the Dev-only configuration, but I don't see why we can't start with a simple tagging/flagging system and extend it later.
And IMO module can 'suggest' to which segment a configuration belongs to, but that is changeable.

I'm working on an approach where the site-builder can switch ON an 'Advance configuration handling' which

  • on Export tags the exported items using the segments described on my session (Primary, Secondary, Initial, Custom i.e. Dev)
  • on Import the system imports/ignores the segments according with their specs (Primary creates/updates/deletes, Secondary keeps untouched, Initial suggests the import, and leaving to the site-builder to decide what to do with the config items for the Custom segments. By default will ignore custom tags.

The change itself shouldn't be that complicated. We can tag the configuration items either with a new property in the schema OR simply by moving the files - on export - in named subfolders.
Then the import process will conditionally handle each item depending by its tag.

The settings for each segment will leave on the State API, so we don't mess up with shipping new settings through the config.

bircher’s picture

Hi gambry

I checked your slides and like the segmentation you made. However, I think you misrepresented config_split because its graylist feature cover the two segments you said it doesn't address. I know, I know it needs more documentation and by design you need to export the configuration before you import it again so that all the configuration is always in git. I worked on a new branch for config_ignore that also uses config_filter (see our blog post about config_filter) so that people who want to use configuration management without seat belts can do so.

I have also thought about tagging the configuration but as I was making a contrib module I didn't want to change the schema for all the configuration because I wanted config_split to be the least invasive as possible. Of course for core we could change that but that changes the scope of the change quite drastically.

So I would keep things relatively simple first.

On the issue itself I have a few other thoughts to share.

I thought about how this could work and I now agree with:

only ask this question when the module is going to be installed

.
Because we would introduce a "development state" and a "production state" in one way or another, ie development modules are installed or not. And then we can just allow installing of modules only when the site is in the development state. Switching between the "states" is akin to synchronizing configuration since we potentially have to create/remove database tables. Allowing installing modules only in the development state means we can just add a checkbox and install the modules as normal modules or as development modules. And in the development state a module can safely be switched from normal to dev and dev to normal as all the things are set up for the module to be used.
That leads us to the next observation: A site in the "production state" has to contain all the configuration also for the development modules, that would allow us to keep synchronizing the configuration from as site in "production state" to one in "development state". This is important since webforms or block placements or panels all use configuration and that might change on a site that has the installing of modules disabled in the UI. But it is also imperative to be able to switch back to the "development state" after testing something while having switched to the "production state" locally.
Adding such a "state" or "mode" or whatever we want to call it would then in a followup allow turning on or off some parts of editing configuration.

So I thought of a solution to the things mentioned above: Add config_filter to core and add a filter that filters the active storage in case the site is in the "production state". Of course this means that we also have to have a real export api that drush and co can use since now they read from the active storage and write to the sync storage for exporting configuration.

seanB’s picture

Besides allowing modules to opt-out for all config, there is also a case where a modules want to exclude some config for export. For instance: The groupmenu module allows creating menu's for groups. In this case the menu's are linked to groups, which means those menu's are basically content.

Allowing the module to define which config to ignore for import/export (in this case the menu's linked to groups) would really help a lot.

seanB’s picture

Just thought of some other use cases for optional config. This could also make sense for contact forms en webforms. The modules could provide a setting allowing administrators to configure if the forms should be ignored on export/import.