Problem/Motivation

Projects namespaced in their own custom profile are a common pattern in Drupal development. In such cases custom modules, themes and libraries are placed inside the custom profile.

Scenario: As a site builder I want to be able to use my custom profile namespace while still being able to import configuration with config_installer.

Steps:

  1. Environment: Drupal 8.1.2, Configuration installer 8.x-1.3.
  2. In the codebase I have a custom profile at profiles/myprofile. At this stage the profile is empty (contains only the myprofile.info.yml file).
  3. Before installing Drupal, I'm adding my custom profile and config sync dir in settings.php:
    $config_directories['sync'] = 'config/sync';
$settings['install_profile'] = 'myprofile';
  4. I'm running drush si config_installer (...)

Everything is OK at this stage, the site installs successfully.

Then I'm starting to add custom modules and themes to profiles/myprofile. Let's say I'm adding a custom theme in profiles/myprofile/themes/mytheme. These custom extensions are creating their own configs and are making other configs depending on them. I'm exporting the config again to sync directory defined above in $config_directories['sync'] and I'm reinstalling the site in the same way. But now config import fails:

Unable to install the mytheme theme since it does not exist.
array_keys() expects parameter 1 to be array, null given in Drupal\Core\EventSubscriber\ConfigImportSubscriber->validateModules() (line 99 of
core/lib/Drupal/Core/EventSubscriber/ConfigImportSubscriber.php).

... (removed error trace for readability)

Unable to install the myprofile module since it does not exist.
The configuration synchronization failed validation.

Proposed resolution

To be defined.

It seems that \Drupal\config_installer\Storage\SourceStorage::read() should exclude also all configuration depending on extensions introduced by profiles and then install them later?

Remaining tasks

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#18 2747179-17.patch12.88 KBalexpott
#17 2747179-17.patch3.79 KBalexpott
#17 13-17-interdiff.txt1.27 KBalexpott
#13 2747179-13.patch10.6 KBplopesc
#13 2747179-13-test-only.patch9.06 KBplopesc
#9 2747179-9.patch17.66 KBplopesc
#9 2747179-9-test-only.patch16.42 KBplopesc
#7 2747179-7.patch1.17 KBplopesc
#7 interdiff.txt763 bytesplopesc
#6 2747179-6.patch1.25 KBplopesc
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

claudiu.cristea created an issue. See original summary.

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
CreditAttribution: claudiu.cristea at Webikon commented
Issue summary: View changes
lukewertz’s picture

lukewertz
He/Him
CreditAttribution: lukewertz at Palantir.net commented

Right, so this becomes an issue when using any profile that includes sub-modules... In my case, Acquia's Lightning cannot be installed with config_installer.

sylvainar’s picture

sylvainar CreditAttribution: sylvainar commented

Still struggling on this. Does any of you found a solution ?

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott at Chapter Three commented

The best solution here is to get #2788777: Allow a site-specific profile to be installed from existing config done. Implementing this in contrib requires us to switch out profiles which makes scanning for modules extremely tricky.

plopesc’s picture

plopesc
he/him
Primary language Spanish
CreditAttribution: plopesc at Cocomore AG commented
Status: Active » Needs review
FileSize
2747179-6.patch1.25 KB

I experienced this problem when trying to use this profile to rebuild the project, but the original profile contains modules in it.

The install process threw an exception - Unable to install the %module module since it does not exist. - because it was not able to find that module.

After some debugging, I found that problem was that ConfigImportSubscriber::getModuleData() calls to system_rebuild_module_data()that relies on $install_state global variable.

So, to bypass this false error, here I'm proposing a patch to swap the value of $install_state['parameters']['profile'] to adapt the profile value and pass this validation.

After the validation, this value is reverted back and the process continue smoothly.

Thank you.

plopesc’s picture

plopesc
he/him
Primary language Spanish
CreditAttribution: plopesc at Cocomore AG commented
FileSize
interdiff.txt763 bytes
2747179-7.patch1.17 KB

Updated version of the patch.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented
Issue tags: +Needs tests

It'd be great to add a test for this.

plopesc’s picture

plopesc
he/him
Primary language Spanish
CreditAttribution: plopesc at Cocomore AG commented
Issue tags: -Needs tests
FileSize
2747179-9-test-only.patch16.42 KB
2747179-9.patch17.66 KB

Attaching the previous patch includiing new tests to prove that works.

While working on this, I had to fix a couple of bugs I found in the module related to Symfony upgrade #2949774: Tarball upload feature is broken due to Symfony upgrade and how to handle module test fixture files #2949778: Remove pre 8.3 tarballs for testing.

In order to test this issue, the 2 mentioned above should be merged before.

Thank you.

hawkeye.twolf’s picture

hawkeye.twolf
they/them or he/him
Primary language English
Location ᏙᎩᏯᏍᏗ Unalatogiyasdi, Tsalaguwetiyi (Cherokee country)
CreditAttribution: hawkeye.twolf at Kalamuna commented

FYI, the patch in number #9 differs from the patch in #6 on one line; missing the declaration of global $install_state. I haven't actually looked at the code to see if that's needed or not. Also, I tried both patches, and neither solved the issue for me when trying to install with a set of configuration exported from a site originally installed with the lightning profile. The configuration files don't pass validation because of lightning.versions.yml and user.role.layout_manager.yml.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented

@plopesc thanks for fixing the tests on 8.4.x! I've committed your patches and fix another thing. Tests are now green on 8.4.x and 8.5.x. Nice one.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented
Status: Needs review » Needs work

There's no 8.3 fixtures anymore since 8.3 is not supported. All the fixtures are for 8.4.x / 8.5.x

plopesc’s picture

plopesc
he/him
Primary language Spanish
CreditAttribution: plopesc at Cocomore AG commented
Status: Needs work » Needs review
FileSize
2747179-13-test-only.patch9.06 KB
2747179-13.patch10.6 KB

Hello, as @derek.deraps mentioned above, I forgot to include one line in the #9 patch. I was working in 3 issues at the same time and messed up my git diffs :)

Here I'm attaching a re-roll of that patch taking into account your last commits and also adding back that missing line.

Regarding fails mentioned in #10, I only have to say that we have this patch working in a couple of real projects where we have custom profiles including nested modules, and tests are green.

Could you please provide more info to try to trace those errors you are still having with lightning?

thank you

hawkeye.twolf’s picture

hawkeye.twolf
they/them or he/him
Primary language English
Location ᏙᎩᏯᏍᏗ Unalatogiyasdi, Tsalaguwetiyi (Cherokee country)
CreditAttribution: hawkeye.twolf at Kalamuna commented

@plopesc, thank you much. I ended up just working around the issue for now (standard site install, set the site UUID, then run a normal config import) but if I find some time to jam on the issue later and recreate it in a vanilla environment, I'll report back. Cheers!

MorinLuc0’s picture

MorinLuc0 CreditAttribution: MorinLuc0 as a volunteer and at Coldfront Labs Inc. commented
Status: Needs review » Reviewed & tested by the community

The issue that @derek.deraps is running into is a lightning specific problem. I received the same error.
See issue 2933445

I applied the lightning patch and this patch and I was able to use the config_installer to install a profile previously installed with lightning.

I have marked the issue RTBC unless there is any other work that needs to be finished for this issue ?

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented
Status: Reviewed & tested by the community » Needs work 
+++ b/src/Tests/ConfigInstallerNestedDependenciesProfile.php
@@ -0,0 +1,48 @@
+class ConfigInstallerNestedDependenciesProfile extends ConfigInstallerTestBase {

This test should also test that 'Nested module' is installed. I'm pretty sure that is it enabled in the db dump but an explicit assertion would also be good. Override \Drupal\config_installer\Tests\ConfigInstallerTestBase::testInstaller() and call the parent but add an assertion about the module being installed.

Also I['m guessing you copied from ConfigInstallerNoDependenciesProfile but actually tests by convention should end in "Test".

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented
Status: Needs work » Reviewed & tested by the community
FileSize
13-17-interdiff.txt1.27 KB
2747179-17.patch3.79 KB

Fixed #16

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented
FileSize
2747179-17.patch12.88 KB

Forgot to diff with --binary.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented

Created @claudiu.cristea for creating the issue.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented
Status: Reviewed & tested by the community » Fixed

All tests pass

Drupal test run
---------------

Tests to be run:
  - Drupal\config_installer\Tests\ConfigInstallerEnSecondTest
  - Drupal\config_installer\Tests\ConfigInstallerFrDirectorySyncTest
  - Drupal\config_installer\Tests\ConfigInstallerFrTarballTest
  - Drupal\config_installer\Tests\ConfigInstallerNoDependenciesProfileTest
  - Drupal\config_installer\Tests\ConfigInstallerSyncTest
  - Drupal\config_installer\Tests\ConfigInstallerTarballTest
  - Drupal\config_installer\Tests\UninstalledProfileModulesTest

Test run started:
  Thursday, March 8, 2018 - 21:53

Test summary
------------

Drupal\config_installer\Tests\ConfigInstallerEnSecondTest     34 passes
Drupal\config_installer\Tests\ConfigInstallerFrDirectorySync  30 passes
Drupal\config_installer\Tests\ConfigInstallerFrTarballTest    33 passes
Drupal\config_installer\Tests\ConfigInstallerNoDependenciesP  26 passes
Drupal\config_installer\Tests\ConfigInstallerSyncTest         33 passes
Drupal\config_installer\Tests\ConfigInstallerTarballTest      37 passes
Drupal\config_installer\Tests\UninstalledProfileModulesTest   41 passes

Test run duration: 1 min 48 sec

  • alexpott committed d1a7a41 on 8.x-1.x 
    Issue #2747179 by plopesc, alexpott, claudiu.cristea: Import config...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Elin Yordanov’s picture

Elin Yordanov CreditAttribution: Elin Yordanov commented

This patch doesn't fix the issue with the themes provided by a profile. I still get the error:

> Unable to install the %theme theme since it does not exist.

Also see the related issue #2975675: Themes provided by profiles not recognized when importing config. May be it should be closed as a duplicate of this issue and this issue should be reopened, since the committed fix is incomplete?

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented

@Елин Й. no need to re-open this issue. We can just note that #2975675: Themes provided by profiles not recognized when importing config is completing the fix. That issue needs a test written.

eelkeblok’s picture

eelkeblok
him/his
Primary language Dutch
Location Netherlands 🇳🇱
CreditAttribution: eelkeblok at One Shoe commented

It looks like there is at least something left here, because I was having trouble installing my custom profile and getting the same error message as mentioned in the summary. I then noticed the patch which removed the known limitation that config_installer will not work with profiles containing modules. Since my profile does include modules, this seemed too much coincidence, so I removed the modules. Lo and behold, the site installed then without any problems.

Edit: Sorry, the submodules were a red herring. Am getting the "can not install [my profile] module because it does not exist", though, not sure why. Very complicated to debug, looking into moving to using the core option directly.