#1697256: Create a UI for importing new configuration introduces a user interface for importing configuration. The interface has been tested with a manual workflow, but not with the git workflow described in comment #93.

Test the patch with the following workflow, and confirm the results. Then, add initial documentation for the workflow on the handbook page linked from the config module's hook_help().

# SETUP DEV SITE
mkdir drupal_dev
cd drupal_dev
wget http://ftp.drupal.org/files/projects/drupal-8.x-dev.tar.gz
tar --strip-components=1 -zxvf drupal-8.x-dev.tar.gz
rm *.gz
git init
git add .
git commit -am "Drupal"

# FIX CONFIG.INFO SO THE PATCH WILL APPLY
vi core/modules/config/config.info.yaml
# Delete everything after "core = 8.x"
git commit -am "Stupid packaging script."

# APPLY THE PATCH
wget http://drupal.org/files/1697256-config-import-63_0.patch
git apply --index 1697256-config-import-63_0.patch
git commit -am "Latest CMI UI patch."

# SETUP SETTINGS.PHP
cd drupal_dev/sites/default
cp default.settings.php settings.php
mkdir files
chmod -R 777 *

# EDIT SETTINGS.PHP AND REPLACE LINE 273 WITH THE FOLLOWING
$config_directories = array(
  CONFIG_ACTIVE_DIRECTORY => array('path' => 'config_xyzzy/dev_to_live'),
  CONFIG_STAGING_DIRECTORY => array('path' => 'config_xyzzy/live_to_dev'),
);

# INSTALL DRUPAL AS YOU SEE FIT

# ADD CONFIG TO GIT
cd config_xyzzy/live_to_dev/
touch README.txt
cd ../../..
git add files
git commit -m "Adding files directory" files
cd ../../..

# SETUP LIVE SITE
git clone file:///Users/gdd/Sites/drupal_dev drupal_live

# SETUP SETTINGS.PHP
cd drupal_live/sites/default
cp default.settings.php settings.php
mkdir files
chmod -R 777 *

# EDIT SETTINGS.PHP AND REPLACE LINE 273 WITH THE FOLLOWING
# NOTE THAT THE ACTIVE AND STAGING DIRECTORIES ARE REVERSED FROM ABOVE
$config_directories = array(
  CONFIG_ACTIVE_DIRECTORY => array('path' => 'config_xyzzy/live_to_dev'),
  CONFIG_STAGING_DIRECTORY => array('path' => 'config_xyzzy/dev_to_live'),
);

# INSTALL DRUPAL AS YOU SEE FIT

# ADD CONFIG TO GIT
git add files
git commit -am "Adding live site config"

Related issues:

Comments

xjm’s picture

Title: Test, improve, and document git workflow » Test, improve, and document git config import workflow
alexpott’s picture

I think one potential issue with proposed git workflow in the issue summary is the UI clears the staging directory due to the code below:

+/**
+ * Form submission handler for config_admin_import_form().
+ */
+function config_admin_import_form_submit($form, &$form_state) {
+  if (!lock()->lockMayBeAvailable(CONFIG_IMPORT_LOCK)) {
+    drupal_set_message(t('Another request may be synchronizing configuration already.'));
+  }
+  else if (config_import()) {
+    // Once a sync completes, we empty the staging directory. This prevents
+    // changes from being accidentally overwritten by stray files getting
+    // imported later.
+    $source_storage = drupal_container()->get('config.storage.staging');
+    foreach ($source_storage->listAll() as $name) {
+      $source_storage->delete($name);
+    }
+
+    drupal_flush_all_caches();
+
+    drupal_set_message(t('The configuration was imported successfully.'));
+  }
+  else {
+    drupal_set_message(t('The import failed due to an error. Any errors have been logged.'), 'error');
+  }
+}

If the directory is under VCS we shouldn't be clearing it here. Here's why we're doing the clearing - http://drupal.org/node/1697256#comment-6641656

PROBLEM I actually skipped this one cos I have no idea how to stage a deletion. Copying all files from dev_active -> prod_staging obviously won't work, because the old YML file will still be sitting there so it'll appear as though there's no change. Seems like we need to clear out the staging dir after import.

I'm not sure how we can handle the conflicting requirements of handling VCS vs FTP controlled deployment. Perhaps the automated deletion should be configurable?

sun’s picture

Category: task » bug
Priority: Normal » Critical

I precisely stated in #1697256-106: Create a UI for importing new configuration that this must not happen. It's beyond my understanding how and why the pasted code of #2 was able to get committed.

That turns this into a critical bug, since Drupal causes data loss and touches data it is not allowed to touch.

heyrocker’s picture

That turns this into a critical bug, since Drupal causes data loss and touches data it is not allowed to touch.

I am having a hard time why this is critical or why this causes data loss. In theory anything arriving in the staging directory came from somewhere else, either through being pulled in via VCS or file-synced manually. Once config is imported its use is done, they have no purpose beyond that action. So what conceptually prevents them from being deleted? In a VCS situation you can just pull again, they're not lost. In the meantime, if they are left in the staging directory, there is a much greater possibility of data being lost for real. Namely:

1) View foo is changed on source and synced to staging on target
2) View is imported on target
3) View is changed on target
4) View bar is changed on source and synced to staging on target
5) View is imported on target
6) Since view foo is still sitting there, it also gets synced and overwrites local changes.

xjm had this happen to her in her testing at #1697256-107: Create a UI for importing new configuration.

All that said, David Strauss brought up that he would like to see a way for this to work in situations where staging is not writeable by the web server user. We discussed some possibilities that would avoid the situation outlined above (mostly centered around keeping timestamps of files and doing hash comparisons) but didn't really come to any conclusions.

heyrocker’s picture

Gave this some thought. I'm on board with making this configurable behavior, as long as the default is to clear the directory. We can add a Settings tab to the import page and put it in there. As I think about it, the kinds of conflict described in #4 is less likely in situations where your deployments are more controlled (aka when you're using a proper deployment mechanism with VCS and a standard workflow.) Also, if we get #1515312: Add snapshots of last loaded config for tracking whether config has changed in, it will further mitigate these conflicts.

xjm’s picture

Priority: Critical » Normal

Yeah, snapshots would at least give us a way to see if the active config has changed locally since the last import, so that we can inform the user what's being synched.

I have no idea how this can be considered a critical bug. I think it's a normal followup task; compromising at major bug despite that I think it's not so much a bug as a point without consensus on the expected behavior.

xjm’s picture

Priority: Normal » Major

Actually compromsing. :)

catch’s picture

Category: bug » task

Compromising again until there's a reproducible bug here.

moshe weitzman’s picture

FYI, we have snapshots and a diff UI now.

funkym’s picture

Testing out the Git workflow with the way I usually set up Drupal sites and I needed to add an extra line to my .gitignore file so I could commit the configuration files but not the whole files directory.

# Ignore paths that contain user-generated content.
sites/*/files/*
sites/*/private/*

# Don't ignore config paths.
!sites/*/files/config_*
funkym’s picture

Here is the process I tested:

  1. Followed the instructions from Building a Drupal site with Git to setup my repository.
  2. Created a .gitignore file from the example.gitignore, and included the extra unignore statement from #10
  3. Installed Drupal
  4. Added, committed and pushed files:
    git add .gitignore
    git add sites/default/files/config_*
    git commit -m "Added .gitignore and configuration files"
    git push
        
  5. Clone out the repository again for development.
  6. Copy settings.php file from Production site to Development site, adjust database config and flip $config_directories['active']['path'] and $config_directories['staging']['path'] variables.
  7. Copy Development site's sites/default/files/config_xyz/active/* files to sites/default/files/config_xyz/staging/*
  8. Development site: add, commit and push changes:
    git add sites/default/files/config_*/staging
    git commit -m "Added staging configuration files"
    git push
        
  9. Pull files on Production site
  10. Note: Here I needed to delete all Production configuration files inside sites/default/files/config_*/staging, I was getting an import error in the UI
  11. Make a configuration change on Development, add, commit and push code.
  12. Pull files on Production site
  13. Navigate to admin/config/development/sync, and Import all
  14. Tada! It works!
alexpott’s picture

It might work but after step 13 what files are left in your production staging directory? I'm guessing not what you would expect...

disasm’s picture

Thanks funkym for getting started on this. A couple of things. It would be useful to have `ls` output for the directory in question both before and after the process. Also, I'm confused why your commiting .gitignore to the repo. Should that go in your your home directory?

xjm’s picture

Also, I'm confused why your commiting .gitignore to the repo. Should that go in your your home directory?

Repo-specific .gitignore files are a common practice with git deployment workflows. The files you want to ignore always everywhere when working on your personal computer are different than the files within a given repository that should never be tracked.

For example, I never want to track the contents of config directories for my various core clones, but I would want to track them in my repository for a specific site.

Gaelan’s picture

Testing.

frob’s picture

Curios, Why did you need to do step 10? that sounds like a bug.

Could the path value be anything for staging/active?
$config_directories['active']['path'] and $config_directories['staging']['path']

Am I correct in assuming that these directories could in fact be anything?

Also, I am a little confused as to why they cannot be set in the UI.

frob’s picture

Issue summary: View changes

config.info is now config.info.yaml

catch’s picture

Issue summary: View changes
Status: Active » Postponed (maintainer needs more info)

Is there anything here that's not covered by #2121751: [META] Making configuration synchronisation work and other issues?

moshe weitzman’s picture

@catch - the issue summary of that issue does not mention Git. We really need to sort out what gets committed where in the use case where a site uses Git and can be deployed to 3 environments (dev/stage/prod). We can do that here or in the other issue, if the scope there is expandable.

alexpott’s picture

Component: config.module » documentation
Status: Postponed (maintainer needs more info) » Needs work

Lets add a page to the configuration document in the handbook that explains a git based workflow.

moshe weitzman’s picture

FYI, we documented this workfkow pretty thoroughly when using a typical dev => stage => prod workflow. See https://www.acquia.com/blog/moving-your-drupal-8-configuration-local-ser...

frob’s picture

I apologize, but I do not consider a video to be documentation.

alexpott’s picture

Status: Needs work » Closed (duplicate)

Closing as duplicate of #1703168: [Meta] Ensure that configuration system functionality matches expected workflows for users and devs and the fact that active config is in the db and D8 export files which can be put into git.

frob’s picture

Status: Closed (duplicate) » Active

This still needs documentation. The referenced issue was about ensuring that the workflow matched what was intended and it was closed because we no longer intend to have it work that way.

Closing because this because the workflow discussed is not longer the way CMI works and people are using CMI successfully today

So we need to document how people are using it today and what the recommended workflow is.
If there is documentation then please link to it and close this issue.

Berdir’s picture

http://drupal.stackexchange.com/questions/85635/what-is-the-suggested-wo... has some information.

Feel free to pick whatever you want from the answers there and create/update d.o documentation.

moshe weitzman’s picture

@frob - there is a blog post linked from that acquia page which is text based. i dont like video based docs either.

beejeebus’s picture

Assigned: Unassigned » beejeebus

i'll try to write something up for this soon.

i spoke about CMI at DrupalSouth recently, and gave a demo of a git-based workflow, so it is all fresh in my mind.

jhodgdon’s picture

documentation++

kim.pepper’s picture

beejeebus++

jhodgdon’s picture

Title: Test, improve, and document git config import workflow » Document git config import workflow
Project: Drupal core » Documentation
Version: 8.0.x-dev »
Component: documentation » Missing documentation

This does not seem like a Drupal Core issue to me. It seems like a "add docs to drupal.org" issue. Accordingly, moving to Documentation project. If there are docs to update that are files in Drupal Core, please update the issue summary and move back to Core.

andypost’s picture

Assigned: beejeebus » Unassigned

Looks a lot of new tools now exists and having at least a stub page would be great