Allow provision-deploy to move a site to a platform with different module paths

omega8cc has opened a similar issue: #1635598: Include Registry Rebuild as a part of Clone and Migrate tasks

+1 from me as this has annoyed me too...

Please mark issues with patches (or sandboxes) as needs review so we don't miss them! :)

I'll close the other issue since it implements a native solution instead of adding another third-party module.

@anarcat: this also calls the third-party module omegacc was referring to.

I've reduced it to one file, ~/.drush/provision_safe_deploy/provision_safe_deploy.drush.inc

With the content:

function drush_provision_safe_deploy_pre_updatedb() {
    drush_invoke_process(d()->name, 'registry-rebuild');
}

I've tried _post_provision_deploy() but that one is called too late in the process.

Some README content could be:
Provision safe deploy
=============

This module runs drush registry_rebuild just before updatedb is executed on a site.
The main benefit of this is seen when a site is cloned from one platform to another. In which case the module paths could be different.

Installation:
This module directory can be placed in ~/.drush/

This module depends on registry_rebuild. Use e.g. 'drush dl registry_rebuild' to install it.

I'm hesitant in adding yet another dependency to Aegir. If we are to implement this, I'd prefer registry rebuild to be merged in directly in Aegir or, better, in drush itself. It's just 200 lines of code after all...

What do you think? Shouldn't we submit this for inclusion in Drush 5?

It would definitely fit better in Drush then in provision.

Let us know when Registry rebuild has been merged into Drush and we can add support into Provision.

Otherwise, people can just install the Registry rebuild extension and follow the approach in #4.

This is not at all clear. Could you please explain this better. For example,

In comment #4
@anarcat: this also calls the third-party module omegacc was referring to. - what third party module - do you mean "registry_rebuild_dev version"?

I've reduced it to one file, ~/.drush/provision_safe_deploy/provision_safe_deploy.drush.inc

Does this mean that we shuold create that file with the content as described as well as download the sandbox project and put it in /var/aegir/.drush/ ?

I have done all this and now cannot clone a site that I used to be able to clone (I just couldn't migrate it (7.14) to a Drupal 7.15 core with modules in a /contrib/ subdirectory - i.e. diferent module paths) - I receive the following 2 warnings, 3 times in the failed task clone log,

include_once(/var/aegir/.drush/provision_safe_deploy/provision/../provision.service.inc): failed to open stream: No such file or directory safe_deploy.drush.inc:3
include_once(): Failed opening '/var/aegir/.drush/provision_safe_deploy/provision/../provision.service.inc' for inclusion (include_path='.:/usr/share/php:/usr/share/pear') safe_deploy.drush.inc:3

Could someone please confirm that I have done the correct steps?

Lastly I believe I am running provision 6.18 or 6.19 (same as aegir) my hostmaster front end is 6.19 - is it necessary to have a dev version of provision, if so, how to do this, please?

I am using drush 4.5 - is that a potential problem?

Thanks for response.

You have clearly added something else to that file, because in the error you're getting to seem to be calling:

include_once('/var/aegir/.drush/provision_safe_deploy/provision/../provision.service.inc')

Or something like that, whereas in #4 the only contents of that file should be:

function drush_provision_safe_deploy_pre_updatedb() {
    drush_invoke_process(d()->name, 'registry-rebuild');
}

without the trailing PHP tag. Sorry that this is a workaround at the moment and we can't really support it much more than that.

The sandbox from wamilton included such a line. Try to reduce it to just that from #4.

Sorry didn't realise that you were using the sandbox, yeah, you shouldn't really need the include or the class in that file to get that working I don't think.

I've just ran into an edge case where this bit me.

On a platform where l10n_update was updated from 7.x-1.0-beta2 to 7.x-1.0-beta3.
The update introduced an extra cache table.
And as the PRE_updatedb hook calls registry-rebuild...
This used the beta3 code to clear the cache, attempting to clear the cache_l10n_update table which will be created during the updatedb process.

I'm not sure if there is a better place to hook into. But wanted to document this anyway.

@helmo

Maybe use registry-rebuild --no-cache-clear, since Provision will clear the caches later in the process anyway. At least this is what we do here: http://drupalcode.org/sandbox/omega8cc/1074910.git/blob/HEAD:/platform/v...

Also, just for reference: #1447272: Incorporate drush rr (Registry Rebuild) into drush as native command

I used the workaround described in #4, but still ran into an issue with ctools plugins whose path was not updated. For now, the only workaround I found for this was, during the migration, to disable the ctools plugin cache, in ctools module directory, file includes/plugins.inc, line 249:

/*if (!empty($cache->data)) {
      // Cache load succeeded so use the cached plugin list.
      $plugins[$module][$type]   = $cache->data;
      // Set $setup to true so we know things where loaded.
      $setup[$module][$type]     = TRUE;
    }
    else {*/
      // Cache load failed so store that we need to build and write the cache.
      $build_cache = TRUE;
    //}

Just to document a use case I have come across where the registry-rebuild is required during a migrate.

I have some sites built on Open Outreach, which includes the RedHen module as an app. Open Outreach 7.x-1.0-rc9 used Redhen 7.x-1.0-beta3, whereas OpenOutreach 7.x-1.0-rc10 used Redhen 7.x-1.0.

A number of files containing classes had been relocated within the directory structure of RedHen when the project was moved from beta3 to official release. Because the registry was holding the old locations of these files, the site was unable to bootstrap on the new platform after a migrate, and migrate fails.

This failure was the result of changes made by module developers and so can not be prevented by site administrators adopting "aegir best practices".

I can confirm that the approach specified in #4 works in this instance.

New features need to be implemented in Aegir 3.x, then we can consider back-porting to Aegir 2.x. Aegir 1.x is essentially deprecated.

While I can't find the reference anymore, I believe the Drush maintainers have rejected the idea of merging registry-rebuild into Drush core. One possible way to deploy this would be to create a hosting_registry_rebuild module, and adding registry-rebuild as a submodule.

Correct, it was discussed here: #1447272: Incorporate drush rr (Registry Rebuild) into drush as native command

[edit] I have linked it already in #15 above.

I'm getting bitten by this almost every time when I'm cloning or migrating sites (e.g. require_once(): Failed opening required '/platforms/newplatform/sites/oldsitename/modules/rules/includes/rules.plugins.inc'). Of course oldsitename does not exist under the new platform.

Adding the following line to drush_provision_drupal_post_provision_deploy seems to do the trick:

provision_backend_invoke(d()->name, 'registry-rebuild --no-cache-clear');

(Although, the no-cache-clear option produces additional warnings.)

@meanderix This is slightly different issue. Or even very different. The issue here is caused by unavoidable path changes in the platform level codebase, while your problem is related to using site level space for code, which is something you should avoid. See this chat for some context: https://hefring.mig5.net/bot/log/aegir/2016-02-22#T610235

@omega8cc Thanks for the link. I've been using Aegir for quite a while now, so I've learned as a rule of thumb to always use site-wide modules. This makes migrations and maintenance so much easier.

However, there are exceptions to the rule, IMO. Sometimes you need to make site-specific customizations (maybe you need to apply certain patches in order to make the module work for a specific use case). This also means you don't want the module to be automatically upgraded. Rather, it's better to manually upgrade the module, when and if required.

In this case it's very useful to keep patched modules locally for specific sites. And, as far as I can tell, this worked fine in 2.x; it's only in 3.x that I'm seeing these errors.

@meanderix It always worked like this, so also in 2.x, and as I pointed out in that IRC chat, Aegir core developer mig5 explained this in 2011. There is no way around this if any module relies via require_once() on some module hidden in the archive. Even invoking registry-rebuild may not help and the task will fail, because Drupal is bootstrapped a few times to various levels within the same Migrate task, not only when the module is extracted from the archive in the new path, and this will populate internal caches with paths changing on the fly, which may get even worse if you use external cache like Redis. But if it helped in your case, it would be interesting to reproduce.

Ok, I had a similar problem with another site:

require_once(/platforms/newplatform/sites/oldsitename/modules/geofield/includes/geofield_backend/default.inc): failed to open stream: No such file or directory plugins.inc:477
require_once(/platforms/newplatform/sites/oldsitename/modules/mailhandler/plugins/ctools/export_ui/mailhandler.inc): failed to open stream: No such file or directory plugins.inc:477

The first error was solved by moving registry-rebuild to before updatedb in drush_provision_drupal_post_provision_deploy. The second error was solved by performing registry-rebuild and cache-clear before the migration process.

@meanderix Feel free to propose patches!

I extracted a patch from #2762701: Aegir 6.x-2.x to 7.x-3.x broken by Drupal 7.50 that fits here...

Registry rebuild would still have to be added to the install process, but this helps when it's available.

I committed this earlier today to be able to test in in Jenkins. Unfortunately it breaks.... registry rebuild might be running to early here for D6 to D7 upgrades.

Compare these two:

• http://ci.aegirproject.org/view/Upgrades/job/U_aegir_6.x-2.x_to_7.x-3.x_...
• http://ci.aegirproject.org/view/Upgrades/job/U_aegir_6.x-2.x_to_7.x-3.x_...

I've reverted it for now.

The patch in #27 helped to overcome #2792713: Drupal 7.50 moved/missing modules. So it's definitely on the right track. That said, a couple other warnings appeared:

[...]
Undefined variable: force_all_cache_clear registry_rebuild.drush.inc:260
[...]
The Drupal caches have NOT been cleared after all registry rebuilds.
It is highly recommended you clear the Drupal caches as soon as possible.
[...]

The first appears to be a minor bug in registry-rebuild. The latter, a warning emitted when the --no-cache-clear option is passed. We'd want to make sure that we can silence that warning, or we'd end up with false positives on all clone and migrate tasks.

Hmm.. The variable seems to be defined, and line 260 is a comment, while the warning has been fixed in head.

We should release new version, I think.

How are we deploying registry-rebuild at present? I see a hosting task (in tasks_extra), but I don't see the back-end. Maybe we should add it as a git sub-module (or sub-tree), so it gets included in hostmaster platform builds?

Rather than invoking the rr commands directly, maybe we should introduce pre- and post-provision-deploy-update hooks, and implement them along side our provision-rr wrapper?

This patch adds the hooks, and related documentation, which looks like what we'd want to implement in hosting_tasks_extra's registry-rebuild wrapper.

We have just released updated rr version: https://www.drupal.org/project/registry_rebuild/releases/7.x-2.4

We still get a lot of warnings with all clones that have site level modules/themes (I have updated registry_rebuild to the latest stable version).
Warnings:

The following theme has moved within the file system: sgk. In order to fix this, clear caches or put the theme back in its original location. For more information, see the documentation page. bootstrap.inc:1128

Include_once(/var/aegir/platforms/d7dev/sites/www.example.com/modules/mimemail/includes/mimemail.mail.inc): failed to open stream: No such file or directory bootstrap.inc:3454

This is of course incorrect as www.example.com was the site name on the d7production platform. Even though we get a bunch of warnings the clone itself does succeed. For now we keep using the solution posted in #2760845

I'm working on the hook implementation in #2839973: Rebuild the registry before database updates are run. If all goes well, I'll RTBC this once I know it works.

I couldn't get that one to work. Any help there/here would be appreciated.

@colan See my comment there.

This hook can't be used within hosting_tasks_extra, for two reasons:

1. You don't want or can't to run the full drush_provision_drupal_post_provision_deploy() there, because it is just too much

2. You can't even access drush_provision_drupal_post_provision_deploy() in contrib, unless you will explicitly include deploy.provision.inc

I have tried to explain this in my comment over there.

Note that while my comment is mainly about the "DB Update" extra task, it applies also to the "Registry Rebuild" extra task.

This seems to have gone dormant but I am still having this issue... what did people do to get things working?

@ShaneOnABike

We applied the following patch for Drupal 7. Note that the first if statement is a crappy Drupal 7 check that you may need to remove or alter.

diff --git a/platform/deploy.provision.inc b/platform/deploy.provision.inc
index bbf8936..860ed78 100644
--- a/platform/deploy.provision.inc
+++ b/platform/deploy.provision.inc
@@ -68,6 +68,22 @@ function drush_provision_drupal_pre_provision_deploy($backup_file) {
       ->succeed('Changed group ownership of temp files in <code>@path

to @gid')
->fail('Could not change group ownership of temp files in @path to @gid');

+ // Add extra statements to database.sql
+ $newpath=d()->site_path;
+ if((strpos($newpath,'d7')!==FALSE) || (strpos($newpath,'drupal_7')!==FALSE) || (strpos($newpath,'hostmaster-7')!==FALSE)) {
+ $sql = '';
+ $old_site = 'sites/' . drush_get_option('old_uri');
+ $new_site = 'sites/' . d()->uri;
+ foreach(array('system','registry','registry_file') as $table) {
+ $sql .= "UPDATE `$table` SET filename = REPLACE(filename,'$old_site','$new_site');\n";
+ }
+ foreach(array('cache','cache_bootstrap','cache_menu','cache_features') as $table) {
+ $sql .= "TRUNCATE `$table`;\n";
+ }
+ file_put_contents(d()->site_path . '/database.sql', $sql, FILE_APPEND);
+ }
+
+
if (drush_get_option('deploy_replace_site', FALSE)) {
$old = d()->site_path . '.restore';
$new = d()->site_path;

Wow thanks very much that moved me back to smooth migrations yahoo!