Using drush 5.0, drupal 7.12, on Linux, drush archive dump (ard) puts two copies of default.settings.php into the archive. This is problematic when drush archive restore (arr) tries to expand the archive.

Reproducable via 1.) drush followed by 2.) drush arr

[vagrant@localhost drupal-7.12]$ drush ard
Archive saved to                                                       [ok]
/home/vagrant/drush-backups/archive-dump/20120330182817/none.20120330_062817.tar.gz
[vagrant@localhost drupal-7.12]$ tar tfvz ~/drush-backups/archive-dump/20120330182817/none.20120330_062817.tar.gz | tail -5
drwxrwxr-x vagrant/vagrant       0 2012-03-30 14:18 drupal-7.12/sites/default/files/styles/
-r--r--r-- vagrant/vagrant   21432 2012-03-30 14:18 drupal-7.12/sites/default/settings.php
-rw-r--r-- vagrant/vagrant   21143 2012-03-30 14:18 drupal-7.12/sites/default/default.settings.php
-rw-rw-r-- vagrant/vagrant     284 2012-03-30 14:28 MANIFEST.ini
-rw-r--r-- vagrant/vagrant   21143 2012-03-30 14:18 drupal-7.12/sites/default/default.settings.php
[vagrant@localhost drupal-7.12]$ drush arr ~/drush-backups/archive-dump/20120330182817/none.20120330_062817.tar.gz
Extracting                                                               [error]
/home/vagrant/drush-backups/archive-dump/20120330182817/none.20120330_062817.tar.gz
using the tar command failed.
Unable to extract site archive tarball to                                [error]
/tmp/drush_tmp_1333132319_4f75fc1f2697f.
[vagrant@localhost drupal-7.12]$ drush arr ~/drush-backups/archive-dump/20120330182817/none.20120330_062817.tar.gz --debug
Bootstrap to phase 0. [0 sec, 3.34 MB]                               [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drush() [0.01 sec, 3.54 MB] [bootstrap]
Cache HIT cid: 5.0-commandfiles-0-5635d6cc7db618aa4f9d889d15578d6b [0.01     [debug]
sec, 3.55 MB]
Bootstrap to phase 0. [0.04 sec, 7.53 MB]                                [bootstrap]
Bootstrap to phase 0. [0.05 sec, 7.53 MB]                                [bootstrap]
Found command: archive-restore (commandfile=archive) [0.05 sec, 7.53 MB] [bootstrap]
Calling chdir(/home/vagrant/drush-backups/archive-dump/20120330182817)       [debug]
[0.07 sec, 7.57 MB]
Executing: tar -C /tmp/drush_tmp_1333132363_4f75fc4b47e7e -xzf none.20120330_062817.tar.gz
  tar: drupal-7.12/sites/default/default.settings.php: Cannot open: File exists
  tar: Exiting with failure status due to previous errors
Calling chdir(/srv/drupal-7.12) [0.24 sec, 7.57 MB]                          [debug]
Calling                                                                      [debug]
_drush_recursive_copy(/home/vagrant/drush-backups/archive-dump/20120330182817/none.20120330_062817.tar.gz,
/tmp/drush_tmp_1333132363_4f75fc4b47e7e/none.20120330_062817HsUSMx.tar.gz)
[0.24 sec, 7.57 MB]
Calling chdir(/tmp/drush_tmp_1333132363_4f75fc4b47e7e) [0.25 sec, 7.58       [debug]
MB]
Executing: gzip --decompress /tmp/drush_tmp_1333132363_4f75fc4b47e7e/none.20120330_062817HsUSMx.tar.gz
Calling chdir(/srv/drupal-7.12) [0.37 sec, 7.58 MB]                          [debug]
Calling chdir(/tmp/drush_tmp_1333132363_4f75fc4b47e7e) [0.37 sec, 7.58       [debug]
MB]
Executing: tar -xf none.20120330_062817HsUSMx.tar
  tar: drupal-7.12/sites/default/settings.php: Cannot open: File exists
  tar: drupal-7.12/sites/default/default.settings.php: Cannot open: File exists
  tar: drupal-7.12/sites/default/default.settings.php: Cannot open: File exists
  tar: Exiting with failure status due to previous errors
Calling chdir(/srv/drupal-7.12) [0.44 sec, 7.58 MB]                          [debug]
Extracting                                                               [error]
/home/vagrant/drush-backups/archive-dump/20120330182817/none.20120330_062817.tar.gz
using the tar command failed. [0.44 sec, 7.58 MB]
Unable to extract site archive tarball to                                [error]
/tmp/drush_tmp_1333132363_4f75fc4b47e7e. [0.44 sec, 7.58 MB]
Command dispatch complete [0.44 sec, 7.56 MB]                               [notice]
Peak memory usage was 8.78 MB [0.44 sec, 7.56 MB]                           [memory]
[vagrant@localhost drupal-7.12]$ drush --version
drush version 5.0

-nick

CommentFileSizeAuthor
#16 drush-archive-dump-1509936-16.patch921 bytesAnonymous (not verified)
#13 drush-test-default-settings-php-is-not-twice-1509936-13.patch1.26 KBjuampynr
#10 drush-archive-dump-symlink-default-settings-php-1509936-10.patch1.49 KBjuampynr
#10 Screenshot-1.png186.77 KBjuampynr
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

filler’s picture

filler CreditAttribution: filler commented

Should have included this:

[vagrant@localhost drupal-7.12]$ tar tfvz ~/drush-backups/archive-dump/20120330182817/none.20120330_062817.tar.gz | grep settings
-rw-r--r-- vagrant/vagrant    753 2012-03-30 14:18 drupal-7.12/themes/garland/theme-settings.php
-rw-r--r-- vagrant/vagrant   4604 2012-03-30 14:18 drupal-7.12/modules/update/update.settings.inc
-r--r--r-- vagrant/vagrant   21432 2012-03-30 14:18 drupal-7.12/sites/default/settings.php
-rw-r--r-- vagrant/vagrant   21143 2012-03-30 14:18 drupal-7.12/sites/default/default.settings.php
-rw-r--r-- vagrant/vagrant   21143 2012-03-30 14:18 drupal-7.12/sites/default/default.settings.php

-nick

filler’s picture

filler CreditAttribution: filler commented

I reproduce on Red Hat Enterprise Linux 6.2 x86_64 (gnu tar 1.23). Meanwhile one of my colleagues reports no issues with this same operation on OSX 10.7.3 (bsdtar 2.8.3 - libarchive 2.8.3): https://gist.github.com/2254161

-nick

filler’s picture

filler CreditAttribution: filler commented

Looks like this passes on my platform @ commands/core/archive.drush.inc : 

237   // Ensure that default/default.settings.php is in the archive. This is needed
238   // by site-install when restoring a site without any DB.
239   // NOTE: Windows tar file replace operation is broken so we have to check if file already exists.
240   // Otherwise it will corrupt the archive.
241   $res = drush_shell_cd_and_exec($workdir, "$tar -tzf %s %s", $destination, $docroot . '/sites/default/default.settings.php');
242   $output = drush_shell_exec_output();
243   if (!$res || !isset($output[0]) || empty($output[0])) {
244     drush_shell_cd_and_exec($workdir, "$tar --dereference -vrf %s %s", $destination, $docroot . '/sites/default/default.settings.php');
245   }

-nick

mattyohe’s picture

mattyohe CreditAttribution: mattyohe commented

Same issue. Same system (RHEL 6.2, gnu tar 1.23)

myohe@ppc:/local/ppc/www/archives/$ lz ppc-Monday-04-09-2012.tar.gz |grep default.settings

Reading directory of  "ppc-Monday-04-09-2012.tar.gz".
ppc/sites/default/settings.php
ppc/sites/default/settings.php~
ppc/sites/default/default.settings.php
ppc/sites/default/default.settings.php

For the time being you can just delete/move that default.settings.php file assuming you're restoring with a DB.

patrickvanelk’s picture

patrickvanelk CreditAttribution: patrickvanelk commented

Same issue on a Ubuntu 8.04 LTS server.

If you chmod +w sites/default and then do a drush archive-dump, drush archive-restore works (there will be only one default.settings.php in the tar). Don't forget to chmod -w both the original folder as well as the restored sites/default folder.

goodale’s picture

goodale CreditAttribution: goodale commented

As a workaround you can fix the archive file by untarring it, and then tarring up the directory, manifest and sql files again. That leaves just the one copy of the file in the archive and drush arr works again.

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented
Status: Active » Postponed (maintainer needs more info)

Please retry with 5.2 or master (soon 5.3)

filler’s picture

filler CreditAttribution: filler commented

I tried with 5.4 and still hit the same buggy behavior. Also 5.4 archive-creates reveal dueling default.settings.php files in the archive. Removing default.settings.php prior to archive-create allows archive-restore to pass.

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented
Version: 7.x-5.0 »
Status: Postponed (maintainer needs more info) » Active

Status change based on last follow-up

juampynr’s picture

juampynr CreditAttribution: juampynr commented
Issue tags: -archive dump ard puts default.settings.php into the archive twice which errors on archive restore arr
FileSize
Screenshot-1.png186.77 KB
drush-archive-dump-symlink-default-settings-php-1509936-10.patch1.49 KB

Drush was attempting to unpack the .tar file using tar -tzf, thus causing the following error:

php > exec('tar -tzf /home/juampy/drush-backups/archive-dump/20120616171424/workshop.20120616_051426.tar workshop/sites/default/default.settings.php 2>&1', $output, $result);
php > print_r($output);                                                                                                                                                 Array
(
    [0] => 
    [1] => gzip: stdin: not in gzip format
    [2] => tar: Child returned status 1
    [3] => tar: Error is not recoverable: exiting now
)

This makes Drush to believe that default.settings.php is not present (line 251 of archive.drush.inc) and therefore the following command gets executed, thus appending twice default.settings.php:

drush_shell_cd_and_exec($workdir, "$tar --dereference -vrf %s %s", $destination, $docroot . '/sites/default/default.settings.php');

Removing the z flag fixes this problem and then the backup can be restored successfully.

I have seen that this line was changed in another commit but the z flag was already present.

There seem to be no test for archive-restore command. I will see how far I can get on helping with that.

juampynr’s picture

juampynr CreditAttribution: juampynr commented
Title: archive dump ard puts default.settings.php into the archive twice which errors on archive restore arr » archive-dump symlinks default.settings.php to itself within sites/default thus causing archive-restore to crash
Priority: Normal » Major
Status: Active » Needs review
Issue tags: +archive dump ard puts default.settings.php into the archive twice which errors on archive restore arr

Restored title to original.

juampynr’s picture

juampynr CreditAttribution: juampynr commented
Title: archive-dump symlinks default.settings.php to itself within sites/default thus causing archive-restore to crash » archive dump ard puts default.settings.php into the archive twice which errors on archive restore arr
Issue tags: +archive dump ard puts default.settings.php into the archive twice which errors on archive restore arr

Doh! Now really reverting the title.

juampynr’s picture

juampynr CreditAttribution: juampynr commented
FileSize
drush-test-default-settings-php-is-not-twice-1509936-13.patch1.26 KB

Here is another patch that adds a test to ensure that default.settings.php has not been added twice.

It currently passes because the method setUpDrupal() installs named sites (dev, stage, prod, etc) while this error occurs when using sites/default. If you replace dev by default at drush_testcase.inc and run the archiveDumpTest again, it will fail with the following error:

There was 1 failure:

1) archiveDumpCase::testArchiveDump
web/sites/default/default.settings.php is not present or has  been added twice to sites/default in the tar.gz file.
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'web/sites/default/default.settings.php'
+'web/sites/default/default.settings.php
+web/sites/default/default.settings.php'

/home/juampy/drush/tests/archiveDumpTest.php:39

FAILURES!
Tests: 1, Assertions: 6, Failures: 1.

However, replacing dev by default breaks other 5 assertions related with core and site alias tests as they expect a single installed site to be configured at sites/dev and not sites/default.

FAILURES!
Tests: 89, Assertions: 513, Failures: 5, Skipped: 2.
moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented
Status: Needs review » Fixed

I removed the 'z' flag as per #10. I don't think the test is really needed anymore, but thanks a ton for authoring it, and diagnosing the problem.

I also wonder if this fixes Windows such that we don't have to even look for default.settings.php but I left this code in place for now.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
Version: » 7.x-5.x-dev
Status: Closed (fixed) » Needs review
FileSize
drush-archive-dump-1509936-16.patch921 bytes

Unfortunately, default.settings.php is still added twice to archive if default site is dumped.

First, it is added when special files in sites/ are added to archive. Next, it is added along with content of default site's site specific directory. Finally, just before archive is compressed, it is ensured that default.settings.php is in the archive (i.e. if it's still not in the archive by this moment it is being added).

Obviously, adding default.settings.php along with special files in sites/ is superfluous and results in default.settings.php being added twice.

I attach patch (compatible with both 7.x-5.x-dev and 8.x-6.x-dev branches) which resolves this issue. Please review and commit. Thank you!

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented
Status: Needs review » Fixed

committed to 5 and 6. thanks.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

SGhosh’s picture

SGhosh CreditAttribution: SGhosh commented

Same issue.

My solution, I did not have the permission to to /tmp. So I issued the command as root and it worked.

Hope this solves the problem for somebody :)