Problem/Motivation

\Drupal\config\Controller\ConfigController::downloadExport() has some code to export all configuration and then put it in an archive. Because this is not abstracted out as a service, Drush, Drupal Console, and any contrib modules wishing to do the same need to re-implement this logic.

Also this prevents improving the general infrastructure around config exports, such as adding events. See also #2991683: Move configuration transformation API in \Drupal\Core\Config namespace.

Proposed resolution

  1. Add a configuration exporter service that can export configuration from a $source storage to a $target storage. This code can be taken from #2991683: Move configuration transformation API in \Drupal\Core\Config namespace
  2. Add an ArchiveStorage that writes configuration into a downloadable archive
  3. Make \Drupal\config\Controller\ConfigController::downloadExport() use the configuration exporter

Comments

tstoeckler created an issue. See original summary.

tstoeckler’s picture

Issue summary: View changes
bircher’s picture

Issue tags: +CMI 2.0 candidate

Adding the CMI 2.0 issue tag.

tstoeckler’s picture

Assigned: Unassigned » tstoeckler

Starting working on this for a bit, nothing to share yet, but wouldn't make sense for anyone to work on this. In case anyone else is motivated to push this forward just let me know here and I will upload a work-in-progress patch.

bircher’s picture

Assigned: tstoeckler » Unassigned

If you are going to use a StorageComparer for this consider reviewing the following issue:
#2993271: Remove unused ConfigManager dependency in StorageComparer

bircher’s picture

Oh! I didn't mean to un-assign you! But there was nothing I could do.

tstoeckler’s picture

Status: Active » Needs review
StatusFileSize
new9.09 KB

Ahh, that's a nice catch!

Here's an initial patch. It is just step 2. from the issue summary, so it provides an archive storage that could then be used together with a configuration exporter by the Config UI. Awesomely, ConfigStorageTestBase exists, so I could actually fix a huge number of bugs and finally verify that it actually works correctly, without ever hooking it up into runtime code. Unfortunately, it's reeeally rough right now, not just around the edges. In particular ::delete() is pretty bad, and the fact that we can't do a proper update, but call out to ::delete() in ::write() is not good. But it does work, so it should allow the rest of the work with the exporter and hooking it all up in the UI to proceed.

In theory, adding the ArchiveStorage could also be split into a separate issue, but not sure what maintainers have to say about that.

So now that that step is done, I think I no longer need to be assigned here. I will try to put some time towards the remaining steps, but this should be a (ugly, but) solid base for anyone to continue work here.

In more concrete terms: I guess the next step would be copying the exporter from #2991683: Move configuration transformation API in \Drupal\Core\Config namespace and then trying to make it work in the UI.

tstoeckler’s picture

StatusFileSize
new8.51 KB
new17.57 KB

Alright, this seems to be all that's needed. And it works for me, as well. Let's see what the bot says.

bircher’s picture

Status: Needs review » Needs work
  1. +++ b/core/lib/Drupal/Core/Config/ArchiveStorage.php
    @@ -0,0 +1,258 @@
    +  public function delete($name) {
    

    yea this is not great. But unfortunately with the ArchiveTar there is not much we can do.
    Maybe we can postpone the ArchiveStorage and implement the exporter without it.

  2. +++ b/core/lib/Drupal/Core/Config/ConfigExporter.php
    @@ -0,0 +1,84 @@
    +  public function export() {
    

    this whole code would be in the first instance just StorageCopier::copyConfig() #3016429: Add a config storage copy utility trait and fix ConfigManager::createSnapshot, but it is where the transformer api will be called.

  3. +++ b/core/modules/config/src/Controller/ConfigController.php
    @@ -19,18 +20,18 @@
    -   * The target storage.
    +   * The active storage.
    ...
    -  protected $targetStorage;
    +  protected $activeStorage;
    ...
    -   * The source storage.
    +   * The synchronization storage.
    ...
    -  protected $sourceStorage;
    +  protected $syncStorage;
    

    I approve of this very much!

  4. +++ b/core/modules/config/src/Controller/ConfigController.php
    @@ -90,16 +102,18 @@ public function __construct(StorageInterface $target_storage, StorageInterface $
       public function downloadExport() {
         file_unmanaged_delete(file_directory_temp() . '/config.tar.gz');
    ...
    -    $archiver = new ArchiveTar(file_directory_temp() . '/config.tar.gz', 'gz');
    +    $archive = new ArchiveTar(file_directory_temp() . '/config.tar.gz', 'gz');
    +    $target = new ArchiveStorage($this->fileSystem, $archive);
         // Get raw configuration data without overrides.
         foreach ($this->configManager->getConfigFactory()->listAll() as $name) {
    -      $archiver->addString("$name.yml", Yaml::encode($this->configManager->getConfigFactory()->get($name)->getRawData()));
    +      $target->write($name, $this->configManager->getConfigFactory()->get($name)->getRawData());
         }
         // Get all override data from the remaining collections.
    -    foreach ($this->targetStorage->getAllCollectionNames() as $collection) {
    -      $collection_storage = $this->targetStorage->createCollection($collection);
    -      foreach ($collection_storage->listAll() as $name) {
    -        $archiver->addString(str_replace('.', '/', $collection) . "/$name.yml", Yaml::encode($collection_storage->read($name)));
    +    foreach ($this->activeStorage->getAllCollectionNames() as $collection) {
    +      $collection_source = $this->activeStorage->createCollection($collection);
    +      $collection_target = $target->createCollection($collection);
    +      foreach ($collection_source->listAll() as $name) {
    +        $collection_target->write($name, $collection_source->read($name));
           }
         }
    

    This is where I think we could improve things already even without the archive storage but by using the active storage instead of the config manager. This would allow us to simply call the transformer first.

    so the code here would be:

      /**
       * Downloads a tarball of the site configuration.
       */
      public function downloadExport() {
        file_unmanaged_delete(file_directory_temp() . '/config.tar.gz');
    
        $archiver = new ArchiveTar(file_directory_temp() . '/config.tar.gz', 'gz');
        
        $storage = $this->activeStorage;
        
        // Get raw configuration data without overrides.
        foreach ($storage->listAll() as $name) {
          $archiver->addString("$name.yml", Yaml::encode($storage->read($name)));
        }
        // Get all override data from the remaining collections.
        foreach ($storage->getAllCollectionNames() as $collection) {
          $collection_storage = $storage->createCollection($collection);
          foreach ($collection_storage->listAll() as $name) {
            $archiver->addString(str_replace('.', '/', $collection) . "/$name.yml", Yaml::encode($collection_storage->read($name)));
          }
        }
    
        $request = new Request(['file' => 'config.tar.gz']);
        return $this->fileDownloadController->download($request, 'temporary');
      }
    

    This way we could replace $storage = $this->activeStorage; with $storage = $this->storageTransformer->transformExport('config.storage.sync'); (or 'config.storage.tarball' etc..)

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

vinay15’s picture

Assigned: Unassigned » vinay15
Issue tags: +DIACWApril2020
vinay15’s picture

Assigned: vinay15 » Unassigned
Status: Needs work » Needs review
StatusFileSize
new3.37 KB
new9.1 KB

Rerolled the patch in #8 and removed the ArchiveStorage and ArchiveStorageTest files as per point 4 in #9.
Also, I tried

  \Drupal::service('config.exporter')->export();

executing in devel and the configuration was exported successfully.

hardik_patel_12’s picture

StatusFileSize
new4.9 KB
new5.02 KB

As @bircher mentioned in point #4 that replace ,

+++ b/core/lib/Drupal/Core/Config/ConfigExporter.php
@@ -0,0 +1,84 @@
+  public function export() {

this whole code would be replaced with static method replaceStorageContents of StorageCopyTrait.

As this export() function is exporting all configuration and then put it in an archive, but what if any contrib modules wishing to export only some set of configuration for example (system.site and user.settings). In that case we can receive set of configuration which needs to be export in export() function as argument as shown in patch.

So now we can export all configuration as well as particular set of configuration or single configuration and then put it in an archive.

bircher’s picture

Status: Needs review » Closed (duplicate)
Related issues: +#3036193: Add ExportStorage to allow config export in third party tools

I think we can close this as a duplicate of #3036193: Add ExportStorage to allow config export in third party tools

While it is not strictly the same, the exporter here would be quite trivial:

namespace Drupal\Core\Config;

class ConfigExporter {
  use StorageCopyTrait;

  protected $export;
  protected $sync;

  public function __construct(StorageInterface $export, StorageInterface $sync) {
    $this->export = $export;
    $this->sync = $sync;
  }

  public function export() {
    self::replaceStorageContents($this->export, $this->sync);
  }
}

with the service definition:

  config.exporter:
    class: Drupal\Core\Config\ConfigExporter
    arguments: ['@config.storage.export', '@config.storage.sync']

See the change notice https://www.drupal.org/node/3037022

I am not sure what a service would be used for other than just calling it from a cli command or maybe in the future from a button in the UI, but then I think we can do that there. #14 suggests that more features would be expected so I think we should open a new issue and scope out the requirements first. The requirements we had planned in this issue have been met.