setup:
* server1, alpha15 (Multiserver management of server1+server2)
* server2, alpha15 (created backups for import to server1); migrating backups to server1 to remove aegir on server2.

siteA hosted on server1

Steps to reproduce:
1) create backup of siteA as from server1.
2) migrate siteA to server2 (set DB as hosted on server2, set PLATFORM as hosted on server2) - WORKS
3) create backup of siteA as from server2 - FAILS. (see log 1)
4) edit siteA, set Domain alias to "sitename.siteB.test.org", SAVE.
5) siteA's content on server2 is 100% REMOVED: (see log2)
server2:/var/aegir/platforms/pressflow-6.19.92/sites/DOMAIN.com.au# du
8 .
6) restore from backup does not work:

LOG 1:
----------------
Log message
Task starts processing
Running: /var/aegir/drush/drush.php @DOMAIN.com.au provision-backup --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @DOMAIN.com.au
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/drushrc.php" into "drupal" scope.
Initialized Drupal 6.19 root directory at /var/aegir/platforms/pressflow-6.19.92
Found command: provision-backup (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_server2homeorg
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Including /var/aegir/.drush/provision/db/backup.provision.inc
Including /var/aegir/.drush/provision/platform/backup.provision.inc
Could not find a Drupal settings.php file at ./sites/default/settings.php.
Site was not found.
An error occurred at function : drush_provision_drupal_provision_backup_validate
Command dispatch complete
Peak memory usage was 5.79 MB
An error occurred at function : drush_hosting_task
Command dispatch complete
Peak memory usage was 12.4 MB
----------------

LOG2:
--------------------
Task starts processing
Running: /var/aegir/drush/drush.php --uri='DOMAIN.com.au' provision-save '@DOMAIN.com.au' --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Found command: provision-save (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @self
Load alias @server_server2homeorg
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Load alias @DOMAIN.com.au
Template loaded: /var/aegir/.drush/provision/provision_drushrc_alias.tpl.php
Changed permissions of /var/aegir/.drush/DOMAIN.com.au.alias.drushrc.php to 600
Generated config Drush configuration file
Changed permissions of /var/aegir/.drush/DOMAIN.com.au.alias.drushrc.php to 400
Command dispatch complete
Peak memory usage was 4.63 MB
Running: /var/aegir/drush/drush.php @DOMAIN.com.au provision-verify --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @DOMAIN.com.au
Found command: provision-verify (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_server2homeorg
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Including /var/aegir/.drush/provision/dns/verify.provision.inc
Including /var/aegir/.drush/provision/platform/backupmigrate/verify.provision.inc
Including /var/aegir/.drush/provision/platform/verify.provision.inc
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/drushrc.php" into "drupal" scope.
Initialized Drupal 6.19 root directory at /var/aegir/platforms/pressflow-6.19.92
Could not find a Drupal settings.php file at ./sites/default/settings.php.
Created directory /var/aegir/platforms/pressflow-6.19.92/sites/DOMAIN.com.au.
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/DOMAIN.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/DOMAIN.com.au/settings.php to www-data
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server server2.home.org.
/var/aegir/platforms/pressflow-6.19.92/sites/DOMAIN.com.au has been synced to remote server server2.home.org.
PDO support available, but the driver has not been installed. Assuming success.
An error occurred at function : drush_provision_drupal_provision_verify_validate
Command dispatch complete
Peak memory usage was 5.43 MB
An error occurred at function : drush_hosting_task
Command dispatch complete
Peak memory usage was 13.05 MB
------------------

CommentFileSizeAuthor
#28 d365687f879a3f346cc44d40e45332e0783b7833.patch.txt2.42 KBomega8cc
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Title: editting site after server migration results in content removal. » db *and* web server migration results in content removal
Priority: Major » Critical
Status: Active » Postponed (maintainer needs more info)

Are you sure the site is still up after step 2? I don't see how doing a backup or verifying the site would destroy data, and I therefore guess that your (rather bold) migrate between both *web* and *db* servers is what fails here. The fact that backup fails is especially suspect in that regard. Can you try again but verify after step 2 to confirm this theory?

omega8cc’s picture

omega8cc CreditAttribution: omega8cc commented
Category: bug » support
Priority: Critical » Normal

This *sounds* like an attempt to "migrate" sites between 2 hostmaster servers? If yes, then it should never be used that way, since the multiserver design requires just one "central" hostmaster while any remote server is just a totally *passive* web and/or db head and not a remote hostmaster.

I think this is what you are trying if you are listing server 1 and server 2 as having alpha15 (both!)

Maybe it is just misunderstanding (wrong usage) and nothing critical? Please correct me if I'm wrong.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Category: support » bug
Priority: Normal » Critical

I think you are correct that the person is trying to migrate between two hostmasters, *but* in itself the target server is not *yet* an hostmaster. I also think this should be supported: worst that can happen is that the "slave" hostmaster would see a site pop into a platform without explanations and it would just pick it up (import)...

Also, we shouldn't assume this is a bug with a slave hostmaster unless we have a confirmation. I don't see what it would impede here anyways...

Good point nevertheless :)

greenscientist’s picture

greenscientist CreditAttribution: greenscientist commented
Category: bug » support
Priority: Critical » Normal

I have a similar problem, but it fails at the step 2 "migration"

I create a site on a platform on Server1-DB1. I try to migrate to a platform on Server2-DB2 and the migration fails.

If I click Retry, I get a "Pass" results, but when I try the web site I get the "Drupal install page".

Probably not relevant, but Server1 is Debian and Server2 is CentOS.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Category: support » bug
Priority: Normal » Critical
Status: Postponed (maintainer needs more info) » Active

So this is a real bug.

Can someone paste a full --debug log of the migration?

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Issue tags: +aegir-0.4

Also, what would help in diagnosing this is testing migrating only to a separate webserver (no db change) and the reverse (same webserver, different db server).

We need to figure out if it's webserver migration, db migration, or both at the same time that's broken. I suspect it's the combination or the DB migration, because we've been migrating sites between webservers without problems, but the db migration hasn't been well tested.

Thanks for the feedback!

greenscientist’s picture

greenscientist CreditAttribution: greenscientist commented

I've done these 4 tests cases (S1 is Server1, D1 is DB 1 and so on)

S1D1 -> S2D1 Task say PASS, but no DB
S1D1 -> S1D2 PASS
S1D2 -> S2D2 Task say PASS, but no DB
S1D1 -> S2D2 Task say PASS, but no DB

BTW, Hostmaster is running on S1

dsobon’s picture

dsobon CreditAttribution: dsobon commented

The reason for hostmaster on SERVER2 is that is the existing setup. I want to move all management to one hostmaster, on SERVER1.

On SERVER2, I manually set apache.conf to include /var/aegir/config/server_SITE2/apache/vhost.d instead of server_master.

After migrating SITEA to SERVER2, this is the result:
* SITEA is hosted successfully on SERVER2, after altering ServerAlias to SITENAME.SERVER2.COM.AU in the apache config, and reloading apache.
* SITEA database is hosted on SERVER2 (confirmed in apache vhost.d config file and mysql client showing the newly create database name 'domaincomau') no problems.
* SERVER1 hostmaster considers this 100% successful and I agree with it.

what doesn't work after migration is:
* CLONE (error: Could not find a Drupal settings.php file at ./sites/default/settings.php. [SITEA on SERVER2 still works])
* BACKUP. (error: Could not find a Drupal settings.php file at ./sites/default/settings.php. Site was not found. [SITEA on SERVER2 still works])
* SITE VERIFY (error: Could not find a Drupal settings.php file at ./sites/default/settings.php. [CONTENT IS PURGED!!! database still there.])
... restore sites/SITEA.COM.AU/
* MIGRATE back to SERVER1 (error: The external command could not be executed due to an application error. [CONTENT IS PURGED; same on both SERVER2 and SERVER2])
... restore sites/SITEA.COM.AU/ on both serves:

* SITEA.SERVER1.COM.AU points to pressflow configuration page ("choose language")
* SITEA.SERVER2.COM.AU works after manual FS-based restore.

dsobon’s picture

dsobon CreditAttribution: dsobon commented

after the failed migration from SERVER2 to SERVER1 (last step) I did the following:
* VERIFY. succeeded! (SITEA.SERVER2.com.au now has a 404, but SITEA.SERVER1.com.au works!!!)
* BACKUP. succeeded!

hostmaster on SERVER1 for SITEA says it is running on platform SERVER2.pressflow-6.19.92, and database SERVER2.com.au.

On SERVER1, I noticed:
* config/server_master/apache/vhost.d/SITEA.com.au is the old config (currently in use), this references SERVER1 (web and DB)
* config/server_SERVER2/apache/vhost.d/SITEA.com.au is the new config (NOT in use), this references: web=SERVER1 and DB=SERVER2

edit: eventually SITEA.SERVER2.com.au worked after trying 10+ minutes later.

dsobon’s picture

dsobon CreditAttribution: dsobon commented

--debug of migration:

Task starts processing
Running: /var/aegir/drush/drush.php @SITE.com.au provision-migrate '@platform_SERVER2pressflow61992' --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/drushrc.php" into "drupal" scope.
Initialized Drupal 6.19 root directory at /var/aegir/platforms/pressflow-6.19.92
Found command: provision-migrate (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_pressflow61992
Including /var/aegir/.drush/provision/db/migrate.provision.inc
Including /var/aegir/.drush/provision/http/migrate.provision.inc
Including /var/aegir/.drush/provision/platform/migrate.provision.inc
Drush bootstrap phase : _drush_bootstrap_drupal_site()
Initialized Drupal site SITE.com.au at sites/SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Putting site under maintenance
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 640
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to www-data
Including /var/aegir/.drush/provision/db/backup.provision.inc
Including /var/aegir/.drush/provision/platform/backup.provision.inc
Drush bootstrap phase : _drush_bootstrap_drupal_configuration()
Adding sites directory to /var/aegir/backups/SITE.com.au-20101116.112405.tar.gz
Temporarily uncloaking database credentials for backup
Undefined index: db_port provision_drupal.drush.inc:141
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 640
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to www-data
Re-cloaking database credentials after backup
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 640
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to www-data
Deleted mysql dump from sites directory
Load alias @platform_SERVER2pressflow61992
Load alias @server_SERVER2
Loading mysql driver for the db service
Loading apache driver for the http service
Running: /var/aegir/drush/drush.php --uri='SITE.com.au' --platform='@platform_SERVER2pressflow61992' --root='/var/aegir/platforms/pressflow-6.19.92' --profile='default' --db_server='@server_SERVER2' provision-save '@SITE.com.au' --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Found command: provision-save (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @self
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Load alias @server_SERVER2
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Template loaded: /var/aegir/.drush/provision/provision_drushrc_alias.tpl.php
Changed permissions of /var/aegir/.drush/SITE.com.au.alias.drushrc.php to 600
Generated config Drush configuration file
Changed permissions of /var/aegir/.drush/SITE.com.au.alias.drushrc.php to 400
Command dispatch complete
Peak memory usage was 6.69 MB
Running: /var/aegir/drush/drush.php --old_uri='SITE.com.au' @SITE.com.au provision-deploy '/var/aegir/backups/SITE.com.au-20101116.112405.tar.gz' --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/drushrc.php" into "drupal" scope.
Initialized Drupal 6.19 root directory at /var/aegir/platforms/pressflow-6.19.92
Found command: provision-deploy (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_SERVER2homeorg
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Including /var/aegir/.drush/provision/db/deploy.provision.inc
Including /var/aegir/.drush/provision/http/deploy.provision.inc
Including /var/aegir/.drush/provision/platform/deploy.provision.inc
Deploying site from /var/aegir/backups/SITE.com.au-20101116.112405.tar.gz
Replacing the existing site at /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au
Granting privileges to SITEACOMAU@SERVER2 on SITEACOMAU
Granting privileges to SITEACOMAU@SERVER2 on SITEACOMAU
Created SITECOMAU database
Running: gunzip -c /var/aegir/backups/SITE.com.au-20101116.112405.tar.gz | tar pxf - in /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore
Successfully extracted the contents of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore
Changed group ownership of files in /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files to www-data
Swapping out the /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore and /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au directories was successful.
Template loaded: /var/aegir/.drush/provision/provision_drushrc_site.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php to 600
Generated config Site Drush configuration file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php to 400
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server SERVER2.
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au has been synced to remote server SERVER2.
Drush bootstrap phase : _drush_bootstrap_drupal_site()
Initialized Drupal site SITE.com.au at sites/SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 640
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to www-data
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server SERVER2.
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au has been synced to remote server SERVER2.
Found a valid version of the block module with schema version 0
Found a valid version of the contact module with schema version 0
Found a valid version of the cookie_cache_bypass module with schema version 0
Found a valid version of the dblog module with schema version 6000
Found a valid version of the filter module with schema version 0
Found a valid version of the help module with schema version 0
Found a valid version of the menu module with schema version 0
Found a valid version of the node module with schema version 0
Found a valid version of the path module with schema version 0
Found a valid version of the path_alias_cache module with schema version 0
Found a valid version of the php module with schema version 0
Found a valid version of the search module with schema version 0
Found a valid version of the statistics module with schema version 6000
Found a valid version of the syslog module with schema version 0
Found a valid version of the system module with schema version 6055
Found a valid version of the taxonomy module with schema version 0
Found a valid version of the user module with schema version 0
Found database dump at /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/database.sql.
Database dump at /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/database.sql is readable
Importing database using command: mysql --defaults-file=/dev/fd/3 SITEACOMAU
Removed symlink for alias /var/aegir/platforms/pressflow-6.19.92/sites/SITEA.SERVER1.com.au
/var/aegir/platforms/pressflow-6.19.92/sites/SITEA.SERVER1.com.au has been removed from remote server SERVER2.
Removed symlink for alias /var/aegir/platforms/pressflow-6.19.92/sites/SITE.otherdomain.com.au
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.otherdomain.com.au has been removed from remote server SERVER2.
Changed permissions of sites/SITE.com.au to 755
Changed permissions of sites/SITE.com.au/themes to 2775
Changed permissions of sites/SITE.com.au/modules to 2775
Changed permissions of sites/SITE.com.au/libraries to 2775
Changed permissions of sites/SITE.com.au/files to 2770
Changed permissions of sites/SITE.com.au/files/tmp to 2770
Changed permissions of sites/SITE.com.au/files/images to 2770
Changed permissions of sites/SITE.com.au/files/pictures to 2770
Changed permissions of sites/SITE.com.au/private to 2770
Changed permissions of sites/SITE.com.au/private/files to 2770
Changed permissions of sites/SITE.com.au/private/temp to 2770
Changed ownership of sites/SITE.com.au/files to aegir
Changed ownership of sites/SITE.com.au/files/tmp to aegir
Changed ownership of sites/SITE.com.au/files/images to aegir
Changed ownership of sites/SITE.com.au/files/pictures to aegir
Changed ownership of sites/SITE.com.au/private to aegir
Changed ownership of sites/SITE.com.au/private/files to aegir
Changed ownership of sites/SITE.com.au/private/temp to aegir
Changed group ownership of sites/SITE.com.au/files to www-data
Changed group ownership of sites/SITE.com.au/files/tmp to www-data
Changed group ownership of sites/SITE.com.au/files/images to www-data
Changed group ownership of sites/SITE.com.au/files/pictures to www-data
Changed group ownership of sites/SITE.com.au/private to www-data
Changed group ownership of sites/SITE.com.au/private/files to www-data
Changed group ownership of sites/SITE.com.au/private/temp to www-data
Removed dump file /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/database.sql after restoring from it
Template loaded: /var/aegir/.drush/provision/http/apache/vhost.tpl.php
Generated config virtual host configuration file
/var/aegir/config/server_SERVER2/apache/vhost.d/SITE.com.au has been synced to remote server SERVER2.
apache on SERVER2 has been restarted
Running: /var/aegir/drush/drush.php @SITE.com.au updatedb --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/drushrc.php" into "drupal" scope.
Initialized Drupal 6.19 root directory at /var/aegir/platforms/pressflow-6.19.92
Drush bootstrap phase : _drush_bootstrap_drupal_site()
Initialized Drupal site SITE.com.au at sites/SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Found command: updatedb (commandfile=core)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_SERVER2
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Drush bootstrap phase : _drush_bootstrap_drupal_configuration()
Drush bootstrap phase : _drush_bootstrap_drupal_database()
Successfully connected to the Drupal database.
Drush bootstrap phase : _drush_bootstrap_drupal_full()
Drush bootstrap phase : _drush_bootstrap_drupal_login()
No database updates required
Including /var/aegir/drush/commands/core/clear.cache.inc
Including /var/aegir/drush/commands/core/clear.cache.inc
Including /var/aegir/drush/commands/core/clear.cache.inc
Including /var/aegir/drush/commands/core/clear.cache.inc
Including /var/aegir/drush/commands/core/clear.cache.inc
Including /var/aegir/drush/commands/core/clear.cache.inc
Including /var/aegir/drush/commands/core/clear.cache.inc
'all' cache was cleared
Finished performing updates.
Command dispatch complete
Peak memory usage was 15.02 MB
Drush bootstrap phase : _drush_bootstrap_drupal_configuration()
Drush bootstrap phase : _drush_bootstrap_drupal_database()
Successfully connected to the Drupal database.
Drush bootstrap phase : _drush_bootstrap_drupal_full()
Changed paths from sites/SITE.com.au to sites/SITE.com.au
Found install profile default
Found 437 modules
Found 6 themes
Cleared all caches
Rebuild node type cache
Rebuild module cache
Rebuild theme cache
Rebuild node access cache
Rebuild menu cache
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/libraries directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/themes/SITE/images directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/themes/SITE directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/themes directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/pictures directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/images/temp directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/images directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/xmlsitemap/40cd750bba9870f18aada2478b24840a directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/xmlsitemap directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/tmp/image directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/tmp directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/resize/images directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/resize directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/backup_migrate/manual directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/backup_migrate/scheduled directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/backup_migrate directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/imagefield_thumbs directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/imagecache/homepage-slider directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files/imagecache directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/files directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/modules directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/private/files directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/private/temp directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore/private directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.restore directory successful.
Content permissions have been rebuilt.
Template loaded: /var/aegir/.drush/provision/provision_drushrc_site.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php to 600
Generated config Site Drush configuration file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php to 400
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server SERVER2.
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au has been synced to remote server SERVER2.
Command dispatch complete
Peak memory usage was 16.32 MB
Running: /var/aegir/drush/drush.php @SITE.com.au provision-verify --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Found command: provision-verify (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_SERVER2
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Including /var/aegir/.drush/provision/dns/verify.provision.inc
Including /var/aegir/.drush/provision/platform/backupmigrate/verify.provision.inc
Including /var/aegir/.drush/provision/platform/verify.provision.inc
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/drushrc.php" into "drupal" scope.
Initialized Drupal 6.19 root directory at /var/aegir/platforms/pressflow-6.19.92
Drush bootstrap phase : _drush_bootstrap_drupal_site()
Initialized Drupal site SITE.com.au at sites/SITE.com.au
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php" into "site" scope.
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 640
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to www-data
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server SERVER2.
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au has been synced to remote server SERVER2.
Drush bootstrap phase : _drush_bootstrap_drupal_configuration()
Drush bootstrap phase : _drush_bootstrap_drupal_database()
Successfully connected to the Drupal database.
Drush bootstrap phase : _drush_bootstrap_drupal_full()
Found install profile default
Found 437 modules
Found 6 themes
Changed permissions of sites/SITE.com.au to 755
Changed permissions of sites/SITE.com.au/themes to 2775
Changed permissions of sites/SITE.com.au/modules to 2775
Changed permissions of sites/SITE.com.au/libraries to 2775
Changed permissions of sites/SITE.com.au/files to 2770
Changed permissions of sites/SITE.com.au/files/tmp to 2770
Changed permissions of sites/SITE.com.au/files/images to 2770
Changed permissions of sites/SITE.com.au/files/pictures to 2770
Changed permissions of sites/SITE.com.au/private to 2770
Changed permissions of sites/SITE.com.au/private/files to 2770
Changed permissions of sites/SITE.com.au/private/temp to 2770
Changed ownership of sites/SITE.com.au/files to aegir
Changed ownership of sites/SITE.com.au/files/tmp to aegir
Changed ownership of sites/SITE.com.au/files/images to aegir
Changed ownership of sites/SITE.com.au/files/pictures to aegir
Changed ownership of sites/SITE.com.au/private to aegir
Changed ownership of sites/SITE.com.au/private/files to aegir
Changed ownership of sites/SITE.com.au/private/temp to aegir
Changed group ownership of sites/SITE.com.au/files to www-data
Changed group ownership of sites/SITE.com.au/files/tmp to www-data
Changed group ownership of sites/SITE.com.au/files/images to www-data
Changed group ownership of sites/SITE.com.au/files/pictures to www-data
Changed group ownership of sites/SITE.com.au/private to www-data
Changed group ownership of sites/SITE.com.au/private/files to www-data
Changed group ownership of sites/SITE.com.au/private/temp to www-data
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 640
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to www-data
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server SERVER2.
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au has been synced to remote server SERVER2.
Template loaded: /var/aegir/.drush/provision/http/apache/vhost.tpl.php
Generated config virtual host configuration file
/var/aegir/config/server_SERVER2/apache/vhost.d/SITE.com.au has been synced to remote server SERVER2.
apache on SERVER2 has been restarted
Cleared all caches
Rebuild node type cache
Rebuild module cache
Rebuild theme cache
Rebuild node access cache
Rebuild menu cache
Content permissions have been rebuilt.
Template loaded: /var/aegir/.drush/provision/provision_drushrc_site.tpl.php
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php to 600
Generated config Site Drush configuration file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/drushrc.php to 400
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server SERVER2.
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au has been synced to remote server SERVER2.
Command dispatch complete
Peak memory usage was 16.19 MB
Dropping database site_1333
Revoking privileges of site_1333@SERVER1 from site_1333
Failed to revoke user privileges
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/libraries directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/themes/SITE/images directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/themes/SITE directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/themes directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/pictures directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/images/temp directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/images directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/xmlsitemap/40cd750bba9870f18aada2478b24840a directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/xmlsitemap directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/tmp/image directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/tmp directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/resize/images directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/resize directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/backup_migrate/manual directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/backup_migrate/scheduled directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/backup_migrate directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/imagefield_thumbs directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/imagecache/homepage-slider directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files/imagecache directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/files directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/modules directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/private/files directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/private/temp directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/private directory successful.
Deleting /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au directory successful.
Command dispatch complete
Peak memory usage was 8.76 MB
Command dispatch complete
Peak memory usage was 16.87 MB

dsobon’s picture

dsobon CreditAttribution: dsobon commented

S1D1 -> S2D1
S1D1 -> S2D2

migration works for both.

But VERIFY after migration FAILS.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Title: db *and* web server migration results in content removal » web server migration results in content removal
Assigned: Unassigned » anarcat

@dboson - you're saying the migrate actually worked if you wanted 10+ minutes, couldn't that be a DNS problem? Keep in mind DNS migration is not implemented yet. (See #922258: update DNS records on inter-server migration for that chunk.) For me the migrate backtrace you give me is very clear: the migrate works, it could possibly be only a DNS problem. For example, during migration you see:

/var/aegir/config/server_SERVER2/apache/vhost.d/SITE.com.au has been synced to remote server SERVER2.
apache on SERVER2 has been restarted

After this, the site should be fully functionnal *ON SERVER2*. Make sure DNS records are correct for that site (ping SITE.com.au) and that the vhost is indeed properly created (apache2ctl -S).

Finally, are you certain the DB is gone? That's the content removal we're talking about here right? Can you check in both database servers to see if there isn't a DB with the right name? Since the DB config sits in the apache vhost, it's possible it's not following the migration path and the site points to the wrong db or DB server.

It seems this is not a DB server migration bug, according to greenscientist and you, all DB-only migration tests work fine (although you mention verify fails, still without providing a debug log ;).

Thanks in advance for the next round of feedback. :) If you're available to talk about this on IRC, i'll be hanging around for next hour or so.

dsobon’s picture

dsobon CreditAttribution: dsobon commented
Title: web server migration results in content removal » db *and* web server migration results in content removal
Assigned: anarcat » Unassigned

DNS is not an issue, because it is INTERNAL DNS, with specific hostnames and addresses. eg:
1) SITEA.SERVER1.local (192.168.x.50)
2) SITEA.SERVER2.local (192.168.x.51)

What I've done now is added the second ALIAS to the frontend _prior_ to migration, to avoid this step of the very small and unrelated problem.

Either way, MIGRATION (web and db) works beautifully.

It's any task run after this MIGRATION that fails completely.

The core problem is this:
* a VERIFY task _after_ MIGRATION (from SERVER1 to SERVER2 successfully) recreates the site on SERVER1 (with only settings.php), then SYNCs the data from SERVER1 to SERVER2, rendering the site on SERVER2 unavailable.

so that's two problems:
1) it recreates the site on SERVER1.
2) it overwrites the site on SERVER2 with the file-based contents on SERVER1.

As per your request, run a BACKUP after MIGRATE:

Log message
------
Task starts processing
Running: /var/aegir/drush/drush.php @SITE.com.au provision-backup --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @SITE.com.au
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/drushrc.php" into "drupal" scope.
Initialized Drupal 6.19 root directory at /var/aegir/platforms/pressflow-6.19.92
Found command: provision-backup (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_SERVER2
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Including /var/aegir/.drush/provision/db/backup.provision.inc
Including /var/aegir/.drush/provision/platform/backup.provision.inc
Could not find a Drupal settings.php file at ./sites/default/settings.php.
Site was not found.
An error occurred at function : drush_provision_drupal_provision_backup_validate
Command dispatch complete
Peak memory usage was 5.79 MB
An error occurred at function : drush_hosting_task
Command dispatch complete
Peak memory usage was 12.4 MB

dsobon’s picture

dsobon CreditAttribution: dsobon commented

I did one small test - and that is to chown -R root.root SITE.com.au on SERVER2. Here is one line log of the VERIFY after the chown test:

/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au could not be synced to remote server SERVER2. Changes might not be available until this has been done. (error: rsync: chgrp
"/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au" failed: Operation not permitted (1) rsync:
delete_file: unlink(var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/themes/SITE/images/vsubitem.gif)

This CONFIRMS my theory that it is syncing from SERVER1 to SERVER2, even though the MIGRATION already did that step with moving the complete site to SERVER2.

dsobon’s picture

dsobon CreditAttribution: dsobon commented

and this one line debug log for a MIGRATE from SERVER2 to SERVER1:
---------------
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au could not be synced to remote server
SERVER2. Changes might not be available until this has been done.
---------------

Which is bizarre to say the least... it's trying to SYNC from SERVER1 to SERVER2 (1->2), even though I
specifically told it to MIGRATE from SERVER2 to SERVER1! (2->1)

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Title: db *and* web server migration results in content removal » web server migration results in content removal

So the problem here is that verify tries to sync data from server1 to server2 while the sites/dir is *not* present on server1. It seems the problem is that the site data doesn't get created properly on the main server.

Verify should create the sites/ directory on SERVER1 so that sync works.

Now, the "reverse sync" problem seems to me like another issue, which we could process separately if you don't mind. :)

Not sure what to do next here, can you provide a debug log of the verify that clears the sites/ dir on SERVER2?

Sorry for just pushing back with requests for more logs, but i'm fairly confused by all this, as I didn't author this code. :)

dsobon’s picture

dsobon CreditAttribution: dsobon commented

debug log for VERIFY
-------------------------------

Log message
Task starts processing
Running: /var/aegir/drush/drush.php --uri='SITE.com.au' provision-save '@SITE.com.au' --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Found command: provision-save (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @self
Load alias @server_SERVER2
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Load alias @SITE.com.au
Template loaded: /var/aegir/.drush/provision/provision_drushrc_alias.tpl.php
Changed permissions of /var/aegir/.drush/SITE.com.au.alias.drushrc.php to 600
Generated config Drush configuration file
Changed permissions of /var/aegir/.drush/SITE.com.au.alias.drushrc.php to 400
Command dispatch complete
Peak memory usage was 4.63 MB
Running: /var/aegir/drush/drush.php @SITE.com.au provision-verify --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @SITE.com.au
Found command: provision-verify (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_SERVER2
Load alias @server_master
Loading mysql driver for the db service
Loading apache driver for the http service
Loading mysql driver for the db service
Loading apache driver for the http service
Load alias @platform_SERVER2pressflow61992
Including /var/aegir/.drush/provision/dns/verify.provision.inc
Including /var/aegir/.drush/provision/platform/backupmigrate/verify.provision.inc
Including /var/aegir/.drush/provision/platform/verify.provision.inc
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/platforms/pressflow-6.19.92/drushrc.php" into "drupal" scope.
Initialized Drupal 6.19 root directory at /var/aegir/platforms/pressflow-6.19.92
Could not find a Drupal settings.php file at ./sites/default/settings.php.
Created directory /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to www-data
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server SERVER2.local.
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au has been synced to remote server SERVER2.local.
PDO support available, but the driver has not been installed. Assuming success.
An error occurred at function : drush_provision_drupal_provision_verify_validate
Command dispatch complete
Peak memory usage was 5.43 MB
An error occurred at function : drush_hosting_task
Command dispatch complete
Peak memory usage was 13.05 MB

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Title: web server migration results in content removal » web server migration results in content removal when site is verified

So just to isolate the issue further here: migrate actually works, it's the verify that destroys the site. It seems some lines here are lying:

Created directory /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.
Template loaded: /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php
Generated config Drupal settings.php file
Changed permissions of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to 440
Change group ownership of /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au/settings.php to www-data
/var/aegir/platforms/pressflow-6.19.92 has been synced to remote server SERVER2.local.
/var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au has been synced to remote server SERVER2.local.

the files do not get created on SERVER1...

I am mostly confused about this, I don't clearly understand how multi-server should work here, so I am not sure I am the best person to diagnose this, and I hope adrian can look at this at some point...

greenscientist’s picture

greenscientist CreditAttribution: greenscientist commented

Should I create a seperate bug report for my problem of invalid user permission?

dsobon’s picture

dsobon CreditAttribution: dsobon commented

Created directory /var/aegir/platforms/pressflow-6.19.92/sites/SITE.com.au.
^^^ created on SERVER1
Generated config Drupal settings.php file
^^^ created on SERVER1

anarcat’s picture

anarcat CreditAttribution: anarcat commented

@greenscientist - yes, that is a good idea.

greenscientist’s picture

greenscientist CreditAttribution: greenscientist commented

Ok, I after bypassing my user issue, I confirm that every actions after a successful migration fails.

(The site is working correctly.)

greenSkin’s picture

greenSkin CreditAttribution: greenSkin commented

We are experiencing the same issue when verifying a site located on server 2, we lose all the files that do not exist on server 1. This is very critical as it results in loss of content uploads from the live site. I do want to note that we are using Aegir 0.4-beta1.

designerbrent’s picture

designerbrent CreditAttribution: designerbrent commented

sub

omega8cc’s picture

omega8cc CreditAttribution: omega8cc commented

If I understand the code correctly, it looks like any remote files are fetched back to the hostmaster only when backup is created, in function drush_provision_drupal_provision_backup():

// synch all filesystem changes back from the remote server.
  d()->service('http')->fetch(d()->site_path . '/files/');

I don't see any similar sync/fetch running on site verify, and remote files are lost because function sync() forces delete without running fetch() first, see below:

  /**
   * If necessary, sync files out to a remote server.
   *
   * @param $path
   *   Full path to sync.
   * @param $additional_options
   *   An array of options that overrides whatever was passed in on the command
   *   line (like the 'process' context, but only for the scope of this one
   *   call).
   */
  function sync($path = NULL, $additional_options = array()) {
    if (!provision_is_local_host($this->remote_host)) {
      if (is_null($path)) {
        $path = $this->config_path;
      }

      if (provision_file()->exists($path)->status()) {
        $options = array_merge(array(
          'relative' => TRUE,
          'keep-dirlinks' => TRUE,
          'omit-dir-times' => TRUE,
          'delete' => TRUE,
        ), $additional_options);


        // We need to do this due to how drush creates the rsync command.
        // If the option is present at all , even if false or null, it will
        // add it to the command.
        if (!isset($additional_options['no-delete'])) {
          $options['delete'] = TRUE;
        }

        if (drush_core_call_rsync(escapeshellarg($path), escapeshellarg($this->script_user . '@' . $this->remote_host . ':/'), $options, TRUE, FALSE)) {
          drush_log(dt('@path has been synced to remote server @remote_host.', array('@path' => $path, '@remote_host' => $this->remote_host)));
        }
        else {
          drush_set_error('PROVISION_FILE_SYNC_FAILED', dt('@path could not be synced to remote server @remote_host. Changes might not be available until this has been done. (error: %msg)', array('@path' => $path, '@remote_host' => $this->remote_host, '%msg' => join("\n", drush_shell_exec_output()))));
        }
      }
      else { // File does not exist, remove it.
        if ($this->shell_exec('rm -rf ' . escapeshellarg($path))) {
          drush_log(dt('@path has been removed from remote server @remote_host.', array('@path' => $path, '@remote_host' => $this->remote_host)));
        }
        else {
          drush_set_error('PROVISION_FILE_SYNC_FAILED', dt('@path could not be removed from remote server @remote_host. Changes might not be available until this has been done. (error: %msg)', array('@path' => $path, '@remote_host' => $this->remote_host, '%msg' => join("\n", drush_shell_exec_output()))));
        }
      }
    }
  }
omega8cc’s picture

omega8cc CreditAttribution: omega8cc commented

Futhermore, the no-delete option is used only once, here:

/**
 * Sync the current Drupal platform and, if applicable, site. Call after
 * finishing operations that affect the filesystem.
 */
function provision_drupal_sync_site() {
  d()->service('http')->sync(d()->root, array('exclude-sites' => TRUE));
  if (d()->type === 'site') {
    d()->service('http')->sync(d()->site_path, array('no-delete' => TRUE));
  }
}

And of course it doesn't help, because "it is called AFTER finishing operations that affect the filesystem."

And there is no no-delete or fetch() used on site verify.

omega8cc’s picture

omega8cc CreditAttribution: omega8cc commented

The more I think about it, the more I feel that the most reliable method would be to always sync /files/ back to hostmaster before running verify with embedded sync(), since in fact every clone/migrate task will always do that anyway, and the only verify task which doesn't need that are site rename, add/remove aliases, and still, this fetch() overhead on every verify would in fact speed up the clone/migrate/backup tasks - less to rsync at once. Otherwise we would need to split the sync() into 2 separate functions to allow using no-delete flag for /files/ because it is not synced in a loop so we could add some if() there, it is just one-path-rsync task.

omega8cc’s picture

omega8cc CreditAttribution: omega8cc commented
Status: Active » Needs review
FileSize
d365687f879a3f346cc44d40e45332e0783b7833.patch.txt2.42 KB

This is untested attempt to resolve this issue. Please try only on test systems!

http://gitorious.org/aegir/provision/commit/d365687f879a3f346cc44d40e453...

omega8cc’s picture

omega8cc CreditAttribution: omega8cc commented

While discussing this on the IRC #aegir, we had an idea to exclude /files/ from the sync() function. That idea was wrong, because doing that, we would break the site delete task probably.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

It seems crazy to me that we do a complete sync of a site just after writing the settings.php, drushrc.php or platform's drushrc.php.

Should those functions have their own simpler sync that only syncs the one file? What happens if your remote server has increased by 3GB of files since the last Verify was run?

And then we could just have a literal 'site' sync (including files) in the pre/post verify hooks or something.. so before a verify is performed we fetch the stuff, and afterward we sync changes back.

I suppose this would need to be on post migrate tasks etc as well, so I'm not sure if this is actually complicating it too much. But it seems a waste of resources and slooooooooow to sync everything just to change a couple of configs.

P.S i just got bitten by this bug myself :) hooray for backups...

mrfelton’s picture

mrfelton CreditAttribution: mrfelton commented

Subscribe.

This just hit me after upgrading from alpha14 (which was working) to beta2.

I just tried to re-verify a remote site, and the result is that all of the data files from the live site have been replaced with the (very old and incomplete) files from the hostmaster.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

It's simpler than all this

provision_drupal_sync_site() does work properly and set no-delete like it should.

The problem in fact is that when we send the options to drush rsync, we are sending delete => TRUE *no matter what*.

      if (provision_file()->exists($path)->status()) {
        $options = array_merge(array(
          'relative' => TRUE,
          'keep-dirlinks' => TRUE,
          'omit-dir-times' => TRUE,
          'delete' => TRUE,
        ), $additional_options);


        // We need to do this due to how drush creates the rsync command.
        // If the option is present at all , even if false or null, it will
        // add it to the command.
        if (!isset($additional_options['no-delete'])) {
          $options['delete'] = TRUE;
        }

So delete is set to TRUE, then we check if no-delete *has not been set* , and if not, we set delete => TRUE again! :)

We never actually set delete = FALSE.

It's my opinion that we should simply remove the first delete => TRUE. The $additional_options stuff will still ensure that delete => TRUE is always set by default, *unless* no-delete has been set, in which case nothing regarding delete is set at all.

So long as Drush's rsync logic does not set --delete by default, this will work.

Checking that now.

omega8cc’s picture

omega8cc CreditAttribution: omega8cc commented

Hmm, this *sounds* logical, but I'm afraid the no-delete option doesn't work no matter what - see the #26 above - the option is set there, yet, remote files get deleted. So, either I don't "click" on something here, or the logic is wrong or at least it is a missing no-delete option in some places.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
Status: Needs review » Fixed

Marking this as fixed, here is the commit. You are welcome to test.

The rule is as follows:

1) If no-delete=TRUE has been set, do not delete
2) If no-delete has not been set, or no-delete has been set to FALSE (no real point in ever doing this, but hey), then delete. (this is thus the default)

Re #33: yes omega8cc that is exactly the problem. The no-delete option *is* set, but it is ignored, because we set delete = TRUE whether or not no-delete was set to TRUE.

askibinski’s picture

askibinski CreditAttribution: askibinski commented

This bug bit us too.
Patch works.

Maybe this page: http://community.aegirproject.org/node/44
could use some more explanation on what exactly happens with rsync?

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

proindustries’s picture

proindustries CreditAttribution: proindustries commented
Status: Closed (fixed) » Active

Guys - I applied the patch against provision.context.server.inc, but I'm still seeing rsync run with a --delete flag.

Will look around and see if I can get a feel for what's causing it...

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
Status: Active » Postponed (maintainer needs more info)

I just can't reproduce it anymore since we fixed it, sorry :s

Executing: rsync -e 'ssh ' -azv --exclude="*.svn*" --stats --progress --relative --keep-dirlinks --omit-dir-times                                                [notice]
'/var/aegir/drupal-6.20/sites/3.mig5-forge.net' 'aegir@web.mig5-forge.net:/' [1.29 sec, 17.78 MB]
  sending incremental file list
  /var/aegir/drupal-6.20/sites/3.mig5-forge.net/settings.php
        3243 100%    2.43MB/s    0:00:00 (xfer#1, to-check=10/17)
         
  Number of files: 17
  Number of files transferred: 1
  Total file size: 35682 bytes
  Total transferred file size: 3243 bytes
  Literal data: 0 bytes
  Matched data: 3243 bytes
  File list size: 409
  File list generation time: 0.001 seconds
  File list transfer time: 0.000 seconds
  Total bytes sent: 476
  Total bytes received: 76

We do multiple rsyncs on site verify for different things. Some of them do set --delete. This is by design.

The one where we must *not* do --delete is on the site dir itself, so that stuff in the site's files dir aren't blown away.

For other stuff, like the platform, and sites/all/ etc, these *do* have --delete set, and this is by design. If this is what you are referring to, you should be putting modules/themes etc on the 'master' copy on the Aegir master server, and then using Verify on the platform to sync these changes to the remote headless web server.

It is not a bidirectional sync (syncing stuff from the remote end back to the master server), except in one special circumstance: that of taking a backup (we sync back the files dir etc). Backups are run on Migrate, Clone, Disable, Enable, Restore and of course Backup itself.

I'm going to leave this open for the moment, as it's a concerning bug and I do *not* want to do another release if this is still a problem for some people. However, I can't reproduce it. I'd like to know what version of Drush etc you are running, what OS, OS version, PHP version etc.

But I do wonder your issue is not about the files/ dir but the general platform, which sometimes surprises people that we kind of 'reset' it back to match what the master server has (this is so we can Migrate safely, with no surprises by previously undetected modules on the remote server).

dsobon’s picture

dsobon CreditAttribution: dsobon commented

The bug I reported happens when you have the same platform path defined on two different servers (MASTER and REMOTE, or REMOTE1 and REMOTE2).

The patch above doesn't resolve the issue I identified but solves a different issue.

See http://drupal.org/node/1039010 for details of bug report attempt #2 and solution.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
Status: Postponed (maintainer needs more info) » Closed (fixed)

Re: #39 yes this has already been covered in that other ticket.

Re: #37, I just built a fresh multiserver installation using HEAD, and I just can't reproduce this bug anymore. No-one else has reported that the files/ data loss bug still persists. And I couldn't solicit any more feedback from you over IRC despite my efforts.

So I'm putting this down to a misconception over what *is* meant to run with --delete rsync flag, which is covered already in our Remote Server / Platform docs on the community site. Closing this.

  • Commit ad0ca56 on dev-migrate_aliases, prod-koumbit, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-subdir-multiserver, 6.x-2.x-backports, dev-helmo-3.x by mig5: 
    Do not send delete = TRUE to the drush rsync call even if no-delete...