Proposal: Configuration Management 2.0 initiative

There's another issue with config management that has not been addressed here:

What to do when a module/theme/distro distributes some default config, which is imported into the site, and then later updates that default config or adds to it. Example: A module provides some views. Later, it adds some columns to the old views, and creates a completely new view. The new view could be added by a hook_update_N(), but the additional columns are not so easy to deal with, because the site manager could have updated that view themselves, so you wouldn't want to overwrite their changes.

There is a contrib (partial) solution to this, which is Config Update Manager https://www.drupal.org/project/config_update -- this module gives you a report (and/or Drush commands) that will show you lists of added, modified, and removed config items, let you look at differences in a meaningful way, and then let you import the new/changed items if you wish. Probably that's about the best you can do, because figuring out what to do programmatically, with no operator input, is unlikely.

There are 4 key core issues related to this, listed on the config_update project page. The big use cases are views and tours that are provided by modules/distros/themes (as well as help topics that are config items, if that module ever gets into core, which will have the same problem as tours).

Also I'd add that the config update manager has a base module that is being used by several other contrib modules, and has some really useful base classes that probably belong in Core as part of the Config API.

I would like to add how we are handling updates in Thunder distribution.

We have joined few parts together in our custom module "thunder_updater" (it's part of distribution code base):

• config_update - we are using it for reverting/importing configuration and also to get configuration lists from active and install/optional storages. In short few services from that module are used. We are not using config differ functionality since we needed 2-way serialization from diff result.
• checklistapi - is used to list all updates provided by the distribution. Every update hook provided by the distribution has an entry in the checklist. Checklist entries are checked/marked from within update hook if update hook is successful or not. For every entry in the checklist, we are providing description what is updated so that user has an overview of changes. If update hook fails, checklist entry will stay unchecked and we have provided a short guide how to change that manually. That means if the configuration is changed on live site in a way, that according to our rules, is not safe to update it - checklist entry will stay unchecked and manual changes have to be done or update can be ignored.

We do also heavy logging. Every config changed by update hook is displayed with success or fail, every installed module is logged, every imported configuration is logged and that is displayed as output for drush command or update.php. Then a user can see for every update hook what has been changed and on top of that, there is checklist entry. We are trying to have updates transparent to users as much as possible.

Additionally, we have Drupal Console command to create an update. That command will generate configuration update definitions, update hook for it and an entry in checklistapi. In that case, workflow to create new configuration update for distribution is quite simple:

1. Make clean install of Thunder
2. Make wanted configuration changes on it
3. Execute provided console command to generate all relevant code for update hook
4. DONE!

Configuration Update Definition (CUD)

CUD is the most important part. Base information to generate it is Diff between two configurations.
Here is one example of CUD:

core.entity_form_display.paragraph.image.default:
  expected_config:
    content:
      field_image:
        settings:
          field_widget_display: rendered_entity
          field_widget_edit: true
  update_actions:
    change:
      content:
        field_image:
          settings:
            field_widget_display: inline_entity_form
            field_widget_display_settings:
              form_mode: inline
            field_widget_edit: false
    remove:
      content:
        field_image:
          settings:
            field_widget_display_settings:
              view_mode: thumbnail

Supported update actions are: change, add and remove. Most important part is expected_config. The expected configuration has to be fulfilled on live site in order to apply configuration update. If the expected configuration is not same, that means a user has changed that part of configuration to their needs and we should not change it.

We still use a module from the distribution since we have some custom parts for entity browser config updates etc., what is not relevant for every distribution or module, but we have extracted base functionality to contrib module and we are planning to switch to that one.
Module is available at: https://www.drupal.org/project/update_helper

We have also tried 3-way diff approach (to merge active and new configuration). In theory 3-way diff sounds great, but it failed in practice really quickly. Unfortunately, I can't remember exact problems, but I know that we abandoned that idea.

I think that most important part is absolute transparency for the user and we should have that on our minds for improvements on configuration management.

P.S. Sorry for the wall of text. :D

That CUD is an interesting idea. Is that generated automatically from the diff, or does it require some other information? If it's just from the diff, please consider adding a feature request to
https://www.drupal.org/project/config_update
that would add a method to create this format on ConfigDiffer. With a pointer to your code, of course. :)

I am interested in this discussion as someone who is not a developer/coder and I live in the beautiful UI. Is this 'new' conclusion not basically what was suggested for discussion by Nedjo years ago http://awebfactory.com/node/536 ? I believe at that time, he was rudely shut down by a few people in key positions to review valuable input from community members. Sometimes great ideas can be dismissed too easily if we do not all take the time to digest them with our critical thinking skills - not to mention the hackers rule "be kind to each other".
We are all here because we love what we do and appreciate help from each other, so apologies can go a long way too - we all realize that we can all be stressed at times and give short shrift to an error gone by...
Let's give credit where it is due - Thanks team, and thanks for the foresight Nedjo!!!

@jhodgdon - regarding Update Helper. Yes, CUD is generated from diff, there is no additional information required. You can take a look at: https://cgit.drupalcode.org/update_helper/tree/src/ConfigHandler.php

I would like to move some parts into config_update, but I didn't have time for that. Now, at least we have the place to talk about it:
#2959072: Normalization should not strip uuid on deeper configuration levels
#2959091: Implement 2-way "format" in config differ

Thanks Angie for sharing the results of your research and focusing them in a positive direction.

I'll share some comments and then some specific technical notes.

• Yes, a CMI 2.0 is strongly needed to address barriers to Drupal 8 usage.
• In the absence of solutions in core, a multitude of approaches have emerged in contrib. A CMI 2.0 stands to learn a lot from that experience. A first step should be outreach to draw in insights and solutions of those active in the work. We're already learning a lot through this issue.
• I'm part of the Drutopia initiative, where improvements on core's config management have been a key focus from the outset--see our 2016 white paper: "a key technical focus of the Drutopia initiative is to build extensions that focus solidly on the free software use case left out of Drupal core: reusable configuration that can be shared across multiple sites." We've done a lot of work that, like that of others, can serve to map out some of the problems and potential solutions.
• Drawing on what's been posted so far, I'd suggest the following as a guiding aim: that, in core, the use case of sharing reusable packages of configuration among multiple sites is on (at least!) a fully equal footing with that of staging configuration on a single site. That aim is not in principle harder to achieve than the staging use case already in core. It's just different. Committing solidly to meeting it could provide the discipline to free the excellent work in CMI from limitations stemming from the focus on staging.
• And big thanks for the support @freescholar. I'm hopeful a fresh (if belated) recognition of issues some of us were raising will be a reminder that space for critical voices is essential for the health of our project. That when we're flagging issues, we're doing so for the good of our collective endeavour.

Okay, on to details. If in core we're going to support and facilitate the use case of extension-provided configuration packages, what problems does that break down into, and how might they be solved?

This question has been my main focus in Drupal development for several years. Here's an attempt to pull together some of what I've learned, found, and contributed to.

Hi,
I very much support this initiative!
Just some comments on the points above:

• 1 config_installer: While the other comments so far point out the problems with distributions and updated configuration of contrib modules, I think installing from configuration is the most dearly missed feature from Core. I would almost argue that the configuration management is broken without it. That said the two issues should make it to core in the same minor version.
• 3b Services per-environment: Don't we already have development.services.yml as a template on how to do that? or what is this referring to?
• 4a config updates for modules that are disabled in the local environment: I agree that this is not documented yet, but if you use Config Split for having the module enabled only on production you can run drush updb && drush csex your-prod-split && drush cim This will export the updated config which is in your prod split and then import it together with the rest in the next step.
• 9 config Ignore UX: we are happy to accept patches to improve the the UX. One solution I recently experimented with is in config_distro_ignore, we could do something similar for config_ignore

Re #4: Yes I see config_update as quite central to that problem space and some of it certainly has a right to become a core API.
Re #5: Thunder has quite a nice approach and I like the CUD idea.
Re #9: A great overview, maybe we can even update the issue summary with your table. It nicely lists the complexity of the different areas of the space and I guess that complexity lead to it not being in core for 8.0

Apropos Config Distro, I just posted a blog post about it with further thoughts on why it was conceived here: http://nuvole.org/blog/2018/apr/09/config-distro.
It would be pretty simple to create a filter for the CUD from update_helper to make them work together.

> Re #9: A great overview, maybe we can even update the issue summary with your table.

Yes. This excellent writeup (and the notes from #10) should definitely be merged into the IS.

@bircher great to see that very timely blog post!

maybe we can even update the issue summary with your table.

Okay, I've taken a first run at updating the IS from comments so far including the table from #9.

I have one small addition to the ideas written already. When building the distro there are some modules that do specific tasks and can be included or not in the beginning. Doing the deGov distro we found it handy to have the reaction on modules that are being installed. Small example that will explain what I mean:

- e.g. we have a module that provides super awesome content type Article with a lot of features and so on
- we don't have meta tags for the articles because we didn't want to add a dependency
- user installs meta tags
- why not to allow our module to react on this event and install the meta tag field/settings for our content type?

This approach could be applied to any module and any configuration change.
We also found that Config rewrite module could be helpful when performing the updates.

I would say it would be nice also to have some special event for updating configs, because such changes are not supposed to be done on updb neither on post updates.

Re #13, also check out Config Selector for additional optional config versions.

And yes Config Distro is meant as the UI to allow site administrators to upate/import distributions configuration. I could imagine that for example as a site admin I install meta tags and then I go to the Config Distro screen and I see an update to the article content type setting up all the settings for it.
Config Distro is just the framework to allow for this workflow, how to do this is up to the distribution and us as the community to figure out the best solution. I could imagine a config rewrite filter that works on distribution updates with Config Distro instead of just when installing a module. I am now not sure if Config Sync already takes that into consideration actually.

I would say it would be nice also to have some special event for updating configs

I've updated the summary to add a row about additional config events. Configuration Update Manager provides a relevant event, ConfigRevertEvent.

I could imagine a config rewrite filter that works on distribution updates with Config Distro instead of just when installing a module. I am now not sure if Config Sync already takes that into consideration actually.

For Configuration Synchronizer, I haven't written integration for Configuration Rewrite, but see Config Actions Provider, which provides parallel integration for Config Actions (which, like Configuration Rewrite, enables programmatic alters).

But come to think of it, Configuration Provider is itself probably just a variation on what's done in a more general way in Config Filter. I've opened #2959928: Consider refactoring to use Config Filter storage and plugins accordingly.

I wonder if the problems of having content dependencies in config (which complicates deployments a lot) is in scope here. As I understand it, this makes things like menus and custom blocks tricky to work with.

I've taken a first pass at converting the proposed enhancements captured so far into issues, using the tag CMI 2.0 candidate. That doesn't directly answer the question of how enhancements might be rolled out, but should provide some material to work with.

I've started to sketch in some relationships between the issues. Next steps could include:

• Identify and tag "CMI 2.0 candidate" other existing issues that are also appropriate for the initiative.
• Add more "CMI 2.0 candidate" issues for what isn't yet captured.
• Go through the issues with "CMI 2.0 candidate" tag and fill in details of how they relate to each other--for instance, what prior issues they require.
• Consider creating overarching issues and assigning existing "CMI 2.0 candidate" issues as children.
• Review, edit, and improve "CMI 2.0 candidate" issues.

I'm going to be offline for the next five weeks, so that's it from me for now.

The issue summary is looking great!

Adding a few related issues that I was aware of...

Copy/paste error one one of the issues.

Tagging so this issue is listed with others.

I think this line in the issue summary

These conclusions validate analysis by Drupal community members in 2015 about barriers posed by the original CMI focus on staging configuration on a single site. See also the 2012 discussion Configuration Management initiative and Drupal distributions.

is not okay. It also points to a 404.

Analysis is all well and good. But as the somewhere people have said we need to analyse why these choices were made. It is all well good to say from the side lines my use-case does not work and you're not looking after my use-case for X reason. But the analysis pointed to here is fundamentally flawed. We focussed on staging a single site because of constraints to do with managing dependencies and not breaking any sites and what was feasible to get Drupal 8 released in the first place. Whatever is done here has to be done with exactly the same care and consideration of data consistency. This is a paramount concern. Reducing that to get desired features in will only make the system more buggy and actually harder and not easier to work with.

It is likely that some of the aims of this initiative will be curtailed because of data consistency and problems when configuration leads to database change. These things are hard and not solved by hand-waving.

@jonathanshaw

I wonder if the problems of having content dependencies in config (which complicates deployments a lot) is in scope here. As I understand it, this makes things like menus and custom blocks tricky to work with.

There's an issue open for the block config dependency on config here: #2756331: Custom blocks cannot be properly exported and imported (may well be other issues too, that was the first one that came up). Those dependencies will affect any kind of distribution/features that attempts to package configuration as much as deployments.

I'm not aware of an issue open for menus. Menu items can be either provided by YAML, configuration, or content entities (via plugins), but a single menu item will not have a dependency on [i]both[/i] configuration and content in the way that block content does. It's true that if you create a content menu item you can't deploy it as configuration, but that's a bit different from the block issue (and applies to any content).

Wanting to deploy configuration and content changes at the same time is something people will want to do and there's not a proper way to do that - but content deployment (even within the same site) is being worked on by the workflow initiative and combining the two won't be viable until that all works (although it could be looked at concurrently).

Today @bircher, @mtodor and I created https://www.drupal.org/project/cmi2 - a drupal project to try gather together the meetings on the new initiative, to act as a starting resource for people wanting to get involved and to coordinate tasks that don't fit into the core issue queue. We had our first meeting just to talk about how we're going to start. Nothing is decided yet at all - see the meeting minutes: https://www.drupal.org/project/cmi2/issues/2968984. Our next meeting in 2 weeks time is probably going to focus on https://www.drupal.org/project/cmi2/issues/2969036 so we have some sort of rationale for why we prioritising what we are. Hopefully we'll have the scoring matrix in place and a rough list of tasks gleaned from this issue before the meeting. All are welcome and all contribution very much appreciated.

but content deployment (even within the same site) is being worked on by the workflow initiative and combining the two won't be viable until that all works (although it could be looked at concurrently).

The general argument seems to be that the config system should be clean of anything that deals with content. The point of that above mentioned issue is, in the case of blocks there is a usecase that muddles the semantics ( is that setting content? Or is it a config?) and it causes a lot of problems for site builders. Please consider including this as part of this initiative.

An issue we would like to address is the combination of a multisite environment with content translation. Our usecase: same codebase, different sites with different default language. Have a look at the following structure of our sites folder with configured languages:

sites
├── default (default language: en – other languages: de)
├── example1.com (en)
├── example2.de (de – en)
├── example3.it (it – en/fr)
├── example4.de (de)
├── …
└── sites.php

Our problem: Neither with configuration management nor with config_split did we get a working workflow up and running.

At the moment we are using Features to export and handle our default config into bundles but this doesn't support multilingual sites (see #2795473) and we are using *.po files to handle translations. But it's often necessary to fix broken translations or configs manually after a new sync (on each instance !!!). And the process depends on the unofficial D8 version of potx. Not really satisfactory.

Any hints to get a working synchronization workflow are appreciated.

After reading the use-case list and thinking about it, a strong inspiration for a path to go comes to my mind:

Let's focus on the concept of ownership. Currently the site ("config/sync") owns the config, and the site user "owns" content. So we have a hardwired connection of storage model and staging/deployment workflow. Other workflows like distributions and merging tools have to "hack" this.

What if we delegate "ownership" of config/content to a dedicated controller of which the current config and content workflows are special cases. Then we can have some config owned by the site, while other config (and maybe content) is owned by the distribution (possibly altered by the site, and merged on updates).

I think this is a frame of thought that can help us step forward.

(More hopefully tomorrow 1345 at CMI 2.0 | Drupal Europe. ;-)

In #1356276-516: Allow profiles to define a base/parent profile @balsama suggested that issue should be postponed because it complicates Drupal's configuration system, which needs work. Most participants in the issue disagree, do people involved in CMI2 have an opinion? If yes, now is the time to speak up ...

Either proposed or approved plan? is the Issue summary up to date?

Updating to reflect that the Configuration installer has been deprecated in favor of Installing Drupal from configuration (only certain profiles)

It sounds like Config Enforce could fit in here somewhere, at least as an example what's solving problems in the wild.

I think this is a duplicate of https://www.drupal.org/about/core/strategic-initiatives-distributions-an... now.

I'm not sure. Looking through the summary, it seems like this was about taking a fresh look at config and taking the underlying API to the next level. The initiative you linked seems to be about providing better starter configuration/profiles/recipes. I could see how updating the config API could help that but those sound like different things and the issues tagged to CMI 2.0 don't seem particularly tied to that initiatives goals.

#35
I would agree with the overall goal that Recipes are currently progressing how config works in drupal core, providing tools 'more like' Features in Drupal7 ecosystem.

This issue certainly does track the way contrib has responed since this issue. Recipes are in my own overview are picking up with this need.

To mention the 'want's of this issue that I would ask around that recipes isn't handling (that i know of!). is ENV based config (config_splits using its 'partial configs') dev/uat/prod codes. Otherwise the goal of having better code side tools and UI's are part of the Recipes outlines..

#34
Yes, this plan 'is stale' at this time. It has great information about how to handle these types of things in d8/d9 sites/lifespan.

Yes, I agree that ENV based config should be easier. Currently, it makes more sense to use settings.php to override config, because then I don't have to manage multiple partial configs.

#37
Perhaps it's possible under the idea that the project would have A Recipe that you enable in that env -- as this depends on Hosting, deploys setups in that hosting. the workflow that team/project uses, etc.

"I would agree with the overall goal that Recipes are currently progressing how config works in drupal core, providing tools 'more like' Features in Drupal7 ecosystem."

Not sure what that means. For my use cases there are better versions of everything in the D8+ ecosystem so I'm just not the target audience.

"To mention the 'want's of this issue that I would ask around that recipes isn't handling (that i know of!). is ENV based config (config_splits using its 'partial configs') dev/uat/prod codes. Otherwise the goal of having better code side tools and UI's are part of the Recipes outlines.."

"Yes, I agree that ENV based config should be easier. Currently, it makes more sense to use settings.php to override config, because then I don't have to manage multiple partial configs."

I use environment driven config _very_ heavily and its pretty straightforward today. Keys module is great if a little rough in the UI. Its designed for "secrets" but the model is allowing the mapping from stores to any config value so mapping from environment to any config value is easy and very powerful and has so far met every need I've had for varying settings per environment with the exception of things like enabling development features locally which still make since to manage with splits.

@neclimdul: Sounds like you adding docs on d.o or a blog post about your approach would help quite a few people be as successful as you with it? ("pretty straightforward today" is the Drupal equivalent of "OMG it is so elegant", right? 😜)

I'm sure there are config issues/features that won't be covered by recipes, but do they need their own special initiative? Nothing stops people working on them.

Use case: as a site administrator, I can install a new site using existing configuration so that I can for example clone new development environments

Solved in core.

Use case: As a site administrator I can review and safely import available configuration updates from installed modules, themes, and the install profile without overriding customizations I've made on the site

Solved by widely used contrib Configuration Update Manager

Use case: As a module developer or site administrator, I can produce a package of configuration that meets a distinct use case

Targeted by Recipes initiative.

I agree with @catch. This issue is a proposal for an initiative. It has not gained traction, and is now outdated. It clutters the issue queue and to focus attention better we should close it as outdated.

Many issues still exist in this area, but as an initiative proposal this should be closed. If there's a new initiative other than recipes that is needed, it would do better to start from scratch as a new more focused proposal.

If there's