Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.Problem/Motivation
As well as #3283827: Support multiple config update paths, which covers what happens after a recipe is applied, there are recipe runtime considerations.
We're trying to give a recipe comprehensive control over the specific configuration it installs. When a recipe is applied, it may attempt to install configuration that's already present on the site, but in a previous (potentially customized) state. How should that work?
Another way to frame this question is: how does it work when a recipe is re-applied?
This is particularly likely as recipes are composable. It's expected that a few commonly used recipes will be reused by many other recipes, meaning it's usual that when a particular recipe is run that some of the recipes it requires will already have been run and configuration provided by the recipe will already exist, albeit potentially in an earlier state.
The questions apply both to extension-provided and recipe-provided configuration.
For example, say that Recipe A is applied. It directly provides several configuration items and also installs several configuration items as provided by Module X.
Time goes by, during which the site builder makes various tweaks to the configuration originally provided by Recipe A and that from Module X.
Now Recipe B is being applied, which requires Recipe A. In the meantime, the configuration Recipe A directly provides has changed a lot. There are new items, some of the old items are no longer provided, and previously existing items are in a changed state. Ditto for the configuration Recipe A selects from Module X--Recipe A now installs some new items from Module X, doesn't select some it used to, and some of the items it still provides are in a changed state (that is, they've been updated since the time the site first applied Recipe A).
What should happen?
With newly newly-provided config it seems straightforward: install it.
Config that used to be provided? Rather than delete it, probably just leave it in the active config.
But for config that was already installed, the case is more complex. Some possibilities:
- Configuration provided by the recipe is left at its current state (as originally installed plus any changes made by the site builder), regardless of whether there have been changes (updates) in the config as now provided.
- Configuration provided by the recipe is updated (reverted) to its current state as provided by Recipe A, losing any customization the site builder may have made.
- Changes to the configuration as provided are merged in and saved to the active config in a way that respects customization.
- The recipe throws a validation exception and is not applied.
The answers may be different for recipe-provided config vs. extension-provided config. Depending on the answers, there may or may not be additional scope not yet captured.
For example, as well as implications for the way config is created or updated, it may be necessary to track which extensions were originally installed via recipes, and which recipes were previously applied on a given site.
Comments
Comment #2
nedjoComment #3
nedjoUpdating to cover recipe-provided config
Comment #4
nedjoFor example, as well as implications for the way config is created or updated, it may be necessary to track which extensions were originally installed via recipes, and which recipes were previously applied on a given site.
Comment #5
nedjoComment #6
alexpottThanks @nedjo for raising this important issue.
I think we should think about the config provided by the recipe as something similar to a set of requirements. If the requirements can be met then we should view the recipe as being successfully applied. So what is happening at the moment is that if the configuration doesn't exist all good... we can let the recipe create it. If the configuration does exist that we'll do a comparison (after removing the UUID, _core and dependencies) if the config matches then we'll allow the recipe to be applied - if not then we throw an exception - see https://git.drupalcode.org/project/distributions_recipes/-/blob/10.0.x/c... - yes we can probably improve how this comparison is done but it is working nicely atm and tested.
I think we should consider in future allowing recipes to be force applied and overwrite configuration in a more destructive manner - and present diffs etc so people will know what's going to change but I view this as stuff for later. I think for now what we have to do is the straghtforward case of not breaking your site and erroring when the requirements are not met. We can deal with the more complex situations later as the requirements for doing that become more apparent.
Comment #7
nedjoYes, there turn out to be a number of wrinkles here that can lead to false negatives when determining if a provided config item is essentially equivalent to a version in the active storage. It's why I wrote the Configuration Normalizer module. There's sort order, of course. That's probably helped by the work in #2852557: Config export key order is not predictable, use config schema to order keys for maps, but there's nothing to require that provided config is in the same order as what's saved to the active storage. There are quirks like the
roleskey in afilter_format, which is present in a provided item but removed on install; see #3007203: Add a plugin for normalizing filter_format config. There are various programmatic changes that are made to specific config types on save and so show up as diffs; see #3007481: Assess and address changes applied at install time. I've been thinking of all this as part of the normalization problem space. That's why, way back when, I opened #2960867: Add a ::normalize() method to config entities to support comparing extension- and recipe-provided to installed configuration (which I recently updated and which may still be as good a place as any to work on this). But now this seems to be a potential fit for config actions. For example, thefilter_formattype could provide an action that removes theroleskey when appropriate (recipe- or extension-provided config is being prepared for comparison to that in the active storage). Does that make sense?I totally understand the attraction and benefits of quick iteration, of roughing things in and circling back. What else have I been doing for the past 20+ years ;) ? But we're all also familiar with the contrary tension. How easy it is for a quick fix to linger, for what's the simplest and most direct to double down on previous patterns and assumptions, to end up taking us further away from rather than closer to a new direction.
My bigger concerns here are:
With composable recipes, we expect, even hope, that the normal workflow will be particular base recipes are routinely re-installed on a given site. We have to also expect that the config in those recipes will evolve over time and that admins will make edits to config as installed. These are not edge cases--they're the core use case.
In D8 the model was that config was installed once then owned by the site. For this initiative to succeed (and here I'll go out on a limb), we need to fundamentally revise that assumption. In theory we're already doing so--updating extension-provided config is explicitly in scope. But - paradoxically - we're focusing all initial work on a model, recipes, that explicitly are not updatable. Except - and this is my hunch in this issue - they actually have to be updatable to serve their own purpose. If a recipe throws an exception the moment any piece of configuration anywhere in the recipe's chain of required recipes has been modified on disk, or even slightly customized by a site owner, we're going to end up with a beautiful system that routinely errors out.
Agreed. And alongside that, an option to merge in config changes while respecting any customizations. In Configuration Synchronizer we do this by snapshotting config as previously provided and then doing a three-way merge (snapshot, current provided version, active).
All this is part of why I'd love to see
ConfigImporterhere rather thanConfigInstaller, and why I opened #3292038: Use ConfigImporter to create/update recipe-provided configuration. It lends itself to these possibilities in waysConfigInstallerdoes not.One by one, we're encountering in recipes pretty much exactly the same challenges as we find in updating extension-provided config--the sub-issues of #2960999: Support updating extension-provided config via configuration synchronization API and UI that I opened years ago. If we solve them generically for recipes, we could get config updates almost for free.
Yes, there's the Update Helper option. There's lots to like there. But - and this is probably my key objection - we need fewer config APIs in core, not more. We have an API for installing config, a whole different one for importing it. And yet a third for updating it? I suspect, by standardizing on
ConfigImporter, we could cover all three use cases and, for extra marks, eventually deprecateConfigInstaller.Going out on another limb--my ideal for the recipe code would be that it actually does little or nothing. Instead, it calls other core config wrangling methods, ones we've long had hacks and workarounds for in contrib but that in the end can only be properly solved in core. For which the new config actions work is an impressive first example.
But - and maybe this is the key point - in the end, all I'm offering here and elsewhere in the initiative is some comments and suggestions. I'm not planning to do much - maybe any - coding. Jobwise I no longer work in Drupal 8+. I'm in wrapup mode. My first Drupal job was at CivicSpace Labs on the first Drupal distribution (I wrote what was probably the first solution set for exporting config to code, a predecessor of Features that I called Packages but that, sadly, CivicSpace declined to open source) and I've focused on distros ever since. This means I've probably worked in Drupal distributions as long as or longer than anyone else. As I prepare to move on from the project, I'm happy to offer what insights I've gained. But it's for others to decide whether and how to take them into account moving forward.
Which I'm happy to be reminded of if at times I seem too stuck or insistent on a supposedly "right" approach ;)
Comment #8
wim leersSubscribing … especially interested in how this touches on conf validation. See my proposal around that in #2164373-28: [META] Untie config validation from form validation — enables validatable Recipes, decoupled admin UIs ….
Comment #9
bircherThere are two cases I think where recipes are "re-applied" one when a composite recipe requires recipes that in turn require the same base recipe. (Site recipe requires two content type recipes which require the editor recipe to add permissions to it)
And secondly at a later stage a recipe is re-applied, but the original recipe the site and the new recipes have are different.
So I am wondering if a recipe should be able to define (maybe automatically) when it is considered "already done". For example if in the Pizza recipe pizza dough is required and tomato sauce is required, but you already have pizza dough in the fridge from when you made focaccia, then applying the pizza dough recipe could just be a no-op.
I don't know if we can reliably determine what a recipe would do without doing it. So if a recipe has evolved and you want to apply it again then for config that is shipped as part of the recipe we can diff it, but for actions we can only apply them (and not fail if it is a no-op like adding a permission that is already existing). But I think for example for the editor recipe if the updated recipe contains more permissions then we wouldn't want to remove the permissions that were added by other recipes via actions. So I don't know what the best suggestion is to make the development of recipes simple. But maybe we can say config in the recipes config directory will just be ignored if it exists already as a default and emit a warning about it.
Comment #10
thejimbirch commentedComment #11
bsnodgrass commentedmoved to Drupal Core