Profile provided configuration entities can have unmeetable dependencies

> This is not an ideal situation as it means we can't really add any new dependency information without breaking install profiles that do this. According to @Berdir we've done this several time over the 8.x dev cycle.

Clarification, what we did several times was breaking customized/overidden config in install profiles for other reasons, usually config schema changes, not specifically by adding new dependencies (I think, can't remember such a case).

Maybe instead of replacing the configuration we can just install the configuration from the module and then when the install profile is instead we can update the configuration entity instead.

Yes, +1. I think that's much more logical. I'm actually surprised it's not how our installer already works. If we do it this way, is there any BC break to contrib/custom install profiles?

We'll have to figure that out. I have two large and complex install profiles that I can test this with, and it would probably make sense to test it with others as well, lightning, thunder, ...

Another possibility is perhaps for each config item in the profile, to install it as soon as all of its dependencies are met? So if user.role.authenticated.yml depends on tour module, then install it right after installing tour module? Not sure if that adds more complexity to the installer than is worth it, but it might be a smaller BC break that way, since the order of config installation wouldn't change for the cases where there aren't dependencies on any modules other than the provider.

Is #2583113: Config import from a different site fails because of shortcut.set.default.yml a duplicate of this?

Please review this issue. I tested Drupal 8.4.2 with custom install profile and works well!

Another use case when config can not be installed, see field.storage.node.body.yml

- The config has a dependency on node and field_permissons modules.
- The node module is always installed before field_permissions module.
- The field.storage.node.body.yml config is installed with node module, which results in:

Configuration objects provided by node have unmet dependencies: field.storage.node.body (field_permissions)

Based on this example it is possible to create a simple test profile which will fail to install (without the patch from this issue).

Started testing this with a large custom install profile.

Was able to move quite a bit of PHP code that changed form/view displays and some other settings to plain config overrides.

I'm also seeing some test fails though, caused by interesting race conditions. Will look more into them, but here's an example:

I'm overriding mailsystem.settings in the install profile. Additionally, swiftmailer_install() adds some additional keys to it in hook_install().

In HEAD, what happens is that mailsystem gets installed, my overridden config gets used, then swiftmailer_install() runs and adds additional things. With the patch, what happens is that mailsystem gets installed with the default config, then swiftmailer_install() runs and adds its stuff and only after that my install profile override kicks in and replaces the whole mailsystem.settings config, including removing the additional keys added by swiftmailer_install(). If it always worked like that then fine, I would have just overriden the final config, but the problem is that this is now changing. It's a change in the installer I suppose, so technically excluded from BC but still. We're introducing something to limit the BC impact of #2571235: [regression] Roles should depend on objects that are building the granted permissions but break a lot of stuff in the process :-/

A similar, more problematic example is system.site, I'm also overriding that to set a few custom settings, but that now means that the generated site UUID is lost. Something is also breaking Ckeditor.

I wonder if the BC break of #14 is reason enough to consider #8?

Though I also do think that the case described in #14 is a bug with the install profile. If the profile includes the installation of swiftmailer, then the overridden config provided by the profile *should* contain the keys needed by swiftmailer. So it's a legitimate question as to whether or not we want to preserve BC for a profile that's not doing so.

> Though I also do think that the case described in #14 is a bug with the install profile. If the profile includes the installation of swiftmailer, then the overridden config provided by the profile *should* contain the keys needed by swiftmailer.

I'm not sure if it's a bug or not, because right now, the install profile can't prevent swiftmailer_install() from acting on it, and depending on what is changed it exactly, already having the changed config might actually result in an unexpected behavior. Maybe not in this case, but there could be others.

Whether or not it is a bug, it is definitely a change, and my point is that we're basically doing this issue to prevent workarounds and BC breaks that #2571235: [regression] Roles should depend on objects that are building the granted permissions would cause. And while the existing workarounds are annoying (this patch allowed me to delete dozens of ugly PHP code in $profile_install()), the advantage of them is that they fail hard, with a pretty clear error message.

The conflicts caused by this are much harder to detect, the installer went through without errors and I likely wouldn't have noticed those problems until much later without my behat tests. My point is basically that I'm not sure if it's worth to make this change, if we just trade some workarounds with others.

Maybe we could support both, by supporting a "postinstall" folder in the install profile? Then you can put things by default in install, and if it fails you have the option to put it there, files there could of course have the exact issues described above, but you only need to use it if you have to it's not something that's suddenly changing in a minor updat for an existing install profile. Sure, it's not a perfect solution, but our install system is far from perfect and *maybe* all those problems will just go away when we only support config import based installers?

Some food for thought on this issue: I had created an install profile by exporting config very close to these instructions https://www.drupal.org/docs/8/creating-distributions/how-to-write-a-drup...

With 8.6, I received these dependencies errors on almost everything until I applied this patch (applied manually on 8.6).

Not sure if this means a totally different method for configuration on an install profile but I even mimicked the demo_umami profile and I can't seem to pin down anything specific that I need to do different.

Is there something I may be missing?

Hello everyone,

I have the problem on the profile I create presently. I updated the default configuration of Image module to add a manual crop. Example of files:

langcode: fr
status: true
dependencies:
  config:
    - crop.type.crop_1x1
  module:
    - crop
name: large
label: 'Large (480×480)'
effects:
  ddd73aa7-4bd6-4c85-b600-bdf2b1628d1d:
    uuid: ddd73aa7-4bd6-4c85-b600-bdf2b1628d1d
    id: image_scale
    weight: -9
    data:
      width: 480
      height: 480
      upscale: false
  7cd03233-541d-472e-851c-de8945ee6709:
    uuid: 7cd03233-541d-472e-851c-de8945ee6709
    id: crop_crop
    weight: -10
    data:
      crop_type: crop_1x1

But despite of I put the configuration files in my optional directory config of my profile, I still have this error when I test my installation profile.

Drupal\Core\Config\UnmetDependenciesException: Configuration objects provided by image have unmet dependencies: image.style.large (crop.type.crop_1x1, crop) in Drupal\Core\Config\UnmetDependenciesException::create() (line 98 of /var/www/html/core/lib/Drupal/Core/Config/UnmetDependenciesException.php).

I test the patch, if the error no longer happens, crop module is correctly installed and configured but the image styles don't include my crop style. So I suppose my optional configurations for image styles aren't applied for default style (these one install by the module image, this is to say large, medium and thumbnail). But it's correctly configured for custom styles I added.

I suppose I need to put this issue in "Need works" like the patch doesn't work as expected in my case.

@Calystod:
Do you have the "crop.type.crop_1x1" config also included in you profile's config/install directory?

Here is a quick patch I've used on our profile to get it up and running on 8.6.1. Will come back and add the tests in, they're missing at the moment.

Patch is mostly a reroll of #2.

@Pasqualle Yes, it's present in the install directory. I do the test to put it in optional directory too, but it doesn't fix the problem.

My crop type is correctly created but not included in the image styles large, medium and thumbnail despite configuration files But it is correctly included in custom styles which aren't by default with the image module)

@DanielVeza Your patch doesn't work with my profile. I have the error of unmeetable dependencies with this one.

I reroll the patch of alexpott for Drupal 8.6.x

It doesn't fix the problem I notice in #23 where a part of my optional configurations aren't applied but the bug of unmeet dependencies doesn't happen with it.

When I will have more time, I will look for the bug I personally met if nobody fix it before.

@Calystod, the problem you describe isn't related to the current issue. Optional configs are being installed at the end of the profile installation, but only those optional configs which haven't been created until that moment. And here is the problem in more details - image.style.large.yml is installed when the image module is being installed, but you've placed this config into the profile's optional config storage. This case is documented in \Drupal\Core\Config\ConfigInstaller::installOptionalConfig():

    $list = array_unique(array_merge($storage->listAll(), $optional_profile_config));
    $list = array_filter($list, function ($config_name) use ($existing_config) {
      // Only list configuration that:
      // - does not already exist
      // - is a configuration entity (this also excludes config that has an
      //   implicit dependency on modules that are not yet installed)
      return !in_array($config_name, $existing_config) && $this->configManager->getEntityTypeIdByName($config_name);
    });

So as the config name already exists it is being ignored.

The solution to your problem is to put the config into the profile's install config storage. If for some reason you cannot do this, then it is better to create a dedicated issue.

---------

+++ b/core/lib/Drupal/Core/Config/ConfigInstaller.php
@@ -270,7 +274,24 @@ protected function getConfigToCreate(StorageInterface $storage, $collection, $pr
+        // Install profiles can provide configuration overrides that have
+        // dependencies that can not possibly be installed. Therefore, only
+        // override simple configuration.

Nice! I didn't thought about simple configuration in the other issue, but the new approach of the current issue solves a problem that I had the last week exactly because of this missing piece in the other patch.

I totally like the current state of the patch here.

@alexpott++

Could you please upload a test only patch? Beside that the issue is RTBC for me!

+++ b/core/modules/config/tests/src/Functional/ConfigInstallProfileUnmetDependenciesTest.php
index 0000000000..1b210736be
--- /dev/null

--- /dev/null
+++ b/core/profiles/testing_config_overrides/config/install/user.role.authenticated.yml

Oh, what is that needed for?

As discussed before in #14-20, I tested this again by applying the patch and comparing the installed configuration, before and after applying the patch.

It still seems to me like we're trading one problem for another. This does indeed allow to put configuration overrides with additional dependencies into config/install instead of requiring code in hook_install(), but by installing them at the end, there are also several cases where dynamic configuration by modules gets overwritten as mentioned there.

Examples:

* The swiftmailer case mentioned in #14. This can be worked around by having the correct configuration upfront, but again, these things are quite hard to detect.

* No uuid in system.site if you have that customized. This is IMHO a pretty big problem, I don't see a workaround for this?

* It is somewhat common to grant auth/anon user roles certain permissions by default in hook_install() of your module, eu_cookie_compliance does that, node in core does, userprotect does and so on. You can easily update your overridden config in your install profile as you are depending on those modules but it's easy to miss if e.g. a new module you add does that.

> system.site overriding is unchanged by the patch above. If you override that you need to have the UUID in it now - this patch does not change that.

Nope, it's not unchanged :)

I have a system.site override in my profile to set some default settings in there. Right now, that's imported when system.module is installed and then system_install() runs and sets the uuid.

With this patch, the default system.site.yml is imported, then system_install() runs, sets the uuid and then later on, my config is imported, overwriting those dynamic keys, resulting in:

$ diff -up config_sync_old/system.site.yml config_sync/system.site.yml 
--- config_sync_old/system.site.yml	2018-10-12 08:30:57.839775204 +0200
+++ config_sync/system.site.yml	2018-10-12 09:17:06.686277539 +0200
@@ -1,4 +1,4 @@
-uuid: 66a1a5da-0358-4f3b-9ed7-38f5d77fbaff
+uuid: ''
 name: Site-Install
 mail: admin@example.com
 slogan: ''

system.site.yml is a simple configuration and as such its configuration override from the profile will be used when the system module is being installed as a part of profile installation. The config is created only once and never touched again by the config installer during profile installation, which is why it is safe to modify simple configuration in the install hook of the module provider of that simple configuration.

It is not surprising that the test doesn't fail as this is the expected behavior for simple configuration, which is not changed by the current patch. The described problem might occur only with config entities - but it is either this or having profiles which cannot be installed at all because of unmet dependencies. The latter can be solved too, but it requires a much more complicated dependency resolution.

#37 sounds like the patch from the duplicate issue, which uses profile overrides for all configs without exception. @Berdir, in what kind of setup are you testing the patch?

I used the patch from #28 because I was testing this on 8.6. @alexpott mentioned that other changes in might have affected this, specifically the recent one about not re-importing configuration from the install profile (#3002655: Optional profile configuration can be unexpectedly recreated after being deleted, which has been backported but is not in a stable release yet)

With that, I'm OK to RTBC this once we have a change record that explains a bit what could change because of this for install profiles. It should specifically mentioned the auth/anon user role and that the config override then trumps all permissions that are dynamically added by modules in hook_install().

Despite this being a bug, it should possibly only be added to 8.7 because it can cause those behavior changes?

Despite this being a bug, it should possibly only be added to 8.7 because it can cause those behavior changes?

👍

+++ b/core/lib/Drupal/Core/Config/ConfigInstaller.php
@@ -270,7 +274,24 @@ protected function getConfigToCreate(StorageInterface $storage, $collection, $pr
       if ($profile_storage->getCollectionName() != $collection) {
...
+      else {
+        // Allow install profiles to override configuration when installing
+        // extension after Drupal installation.
+        $data = $profile_overrides + $data;
+      }

This documentation is not entirely clear to me.

The use case for this is for the deployment of an update to an existing site that includes a new module.
The updated code containing the overrides for the new module would be deployed to the server and then the module would be enabled in a post-update hook.

Maybe something like this would be clearer?

        // Allow install profiles to provide overridden configuration for new
        // extensions that are being enabled after Drupal has already been
        // installed. This allows profiles to ship new extensions in version
        // updates without requiring additional code to apply the overrides.

Will you be preparing for 8.7.0?

Is anybody else having issues with this patch? We use it in our profile and it appears it is no longer working - Although it does apply.

Strange thing, is that it works on PHP7.0 - Installing completes, anything above that and it no longer works. It's bizarre.

For future people - this was the cause https://www.drupal.org/project/drupal/issues/3045349#comment-13054710

The patch from #38 worked for me.

It applied applied cleanly to 8.8.x-dev which I tested on.

Thanks!

I confirm what patch #38 fix UnmetDependenciesException error with my installation profile.

If you cannot provide some config on installation be cause it conflicts with installer, you should be able to override the settings after installation. For example, a hook that triggers after installation would be ideal if I want to standardize some values in `update.settings`.

Alternatively, the installer should merge the default configurations with the existing one without overwriting.

Added a paragraph on specific suggestions and commands that people can use to diff and copy their install profile config. A bit uncommon I guess for a change record, but it's a bit an untypical change record anyway and I did the testing/research with that and thought it might help someone :)

As I said in #40, RTBC now that we have a change record. #55 passed, #56 is just documentation changes.

Off-topic: I still don't really get how default_config_hash stuff is supposed to work exactly. All my files changed on that, even if nothing else changed, I suppose because the uuid was added. But should the default config now have that or not.. and if not, then what's the point? So many questions :)

+++ b/core/lib/Drupal/Core/Config/ConfigInstaller.php
@@ -271,7 +275,26 @@ protected function getConfigToCreate(StorageInterface $storage, $collection, $pr
       }
-      $data = $profile_storage->readMultiple(array_keys($data)) + $data;
+      $profile_overrides = $profile_storage->readMultiple(array_keys($data));
+      if (InstallerKernel::installationAttempted()) {
+        // Install profiles can provide configuration overrides that have
+        // dependencies that can not possibly be installed. Therefore, only
+        // override simple configuration. As this method is called before
+        // modules are installed the only way to determine if they are
+        // configuration entities is the presence of a dependencies key.
+        foreach ($profile_overrides as $name => $override_data) {
+          if (!isset($override_data['dependencies'])) {
+            $data[$name] = $override_data;
+          }
+        }
+      }
+      else {
+        // Allow install profiles to provide overridden configuration for new
+        // extensions that are being enabled after Drupal has already been
+        // installed. This allows profiles to ship new extensions in version
+        // updates without requiring additional code to apply the overrides.
+        $data = $profile_overrides + $data;

Neither of these comments explain where profile overrides of configuration entities get applied. I understand from the issue that it's when the profile is installed, but it's not clear from reading the patch.

That's much better thank you. Given it's only a docs change that I requested when the patch was RTBC, committing from needs review.

Committed 959f175 and pushed to 8.8.x. Thanks!

Nice work pushing the over the line team :). Another patch we can take out of our profile, always a win!

The release note here could use a little work to clarify what the disruption is: What was it like before, and what is it like now? In particular:

Profile configuration should be as the profile author wants it to be at the end of the installation.

Does this mean that the API is supposed to do this now, or does this mean that a profile author (or module author, or someone) would need to make changes in order to accomplish this so that the fix from this issue works? Or something else?

See https://www.drupal.org/issue-summaries#release-notes for more info and examples on what people need from the release notes.

This sentence is still equally unclear though:

Configuration provided by the profile should be as the profile author wants it to be at the end of the installation.

The word "should" is ambiguous. Does it mean "we think the API will handle this for you, but YMMV"? Or does it mean, "You need to make changes to your profile's configuration"? Direct instructions will help. Something like:

Previously, configuration could be altered during profile installation. For $reasons, it is now required that configuration provided by the profile contain the final state in the exported configuration in the module. If your module alters configuration during installation, you must update it to the final state and remove any XYZ implementations that altered it previously. See the change record for suggestions on how to check your configuration and update it.

The wording could use some work still and maybe that isn't accurate but that's the kind of information we need. "Should" has multiple meanings depending on the reader's assumptions.

Also where does this change leave config installation that needs to be dynamic? Or can modules still dynamically alter config? It'd be good to clarify that.

Proposed update:

Previously, modules could make changes to configuration a profile is overriding during their install. This has changed in 8.8 and profile configuration overrides are now installed at the end of an installation. Any profiles that previously modified their configuration during installation must update their configuration so that it contains the final configuration that is desired at the end of installation. Refer to the change record for more information and suggestions for updating your profile's configuration.

It's much easier for people to understand in this order: before -> after -> what they need to change as a result -> where to get more info.

However, this still doesn't explain what to do about individual modules that may modify configuration during installation and how profile owners are supposed to handle that, or whether modules are affected or not. We should add that information to the release note when possible so that the alpha notes can be updated to clarify this. (And also validate that what I wrote above is actually accurate.)

Wait, does this mean my profile submodules can no longer grant permissions to a certain role when they get installed during site installation, as the profile will in the end just overwrite it again with its base config for that role?

Nope, as I submitted it hit me, those submodules, and the form where users can pick 'em, happens later, after the pure "drupal install" step has already finished. Derp.

i have an error at drupal installation with my custom profil - version 2. i have successfully created a custom profile - version 1 - and installation is ok, thus i know how to do it and it works.

Note: i am installing the Drupal files with composer.

EDIT: i've found the problem, in the core.extension.yml the theme category is named theme , but in the YourThemE.info.yml it's named themes.