Merge Provision extensions into Hosting modules

I'm afraid this may cause Catch-22 situations. What if the frontend upgrade will fail, and the paths will get out of sync? How will you handle hostmaster (failed) upgrade rollback, if Aegir backend will get 'moved on the fly' along with hostmaster codebase? Sure, having backend and frontend separate makes things a bit harder to manage, but it also makes them far easier to manage when something will fail during upgrade.

While adding drush integration in contrib modules is a standard practice, we shouldn't just look for convenience, because decoupling backend and frontend, so Drush extensions stay in the command line space and frontend in the installation profile space makes a lot of sense and feels logical, at least to me.

Or maybe it could even improve upgrades reliability and ability to rollback? Because right now we replace Provision *in place* first, so we can rollback automatically only the Hostmaster part.

If we merge Provision into Hostmaster (or rather into the Hosting module), we will be able (or rather forced) to rollback both the backend and frontend -- at least in theory, because in practice the rollback is managed by the currently running (which one at which point?) backend, so we can't really switch it while it runs the rollback, but if it will run the rollback, we will end up also with old Provision again, because we will no longer replace it *in place* separately from running Hostmaster upgrade, so it will become a moving target, making rollbacks in fact rather unpredictable, especially on major upgrades.

This would require some intended crash-tests to make sure we will not make things unexpectedly worse if the rollback is needed, I guess, because I'm not sure we could call it a safe rollback any longer.

The subtree merge looks good.
+1 for reducing the number of repositories.

Would it be an idea to call the new dir 'drush'? As they are tipically just drush extensions.

Or would that conflict with the default loading of drush fil?

This will change the current behaviour where we upgrade Hostmaster using upgraded/replaced first Provision. We will be forced to always use the old backend during the upgrade. This may cause serious problems, because we can't guarantee that different versions of backend and frontend will always cooperate without some inter-connected requirements/assumptions, like, say, compatible Drush version.

What if the upgrade will require sync-ed changes both in the backend and in the fronted, plus there will be Drush version involved which is not compatible with older backend? Are you sure it will work for situations like the upgrade from Aegir 2.x to 3.x, where we not only change the Drupal major version in the frontend, but we also switch Drush version? (yes, I know we discuss changes in 3.x here)

The current steps of our upgrade script are:

1. update drush
2. update provision
3. start hosting-migrate task, which builds a new hostmaster platform and migrates the site there.

We currently do nothing with contrib provision extensions...
So an incompatible extension could already spoil the upgrade, and users have to update those themselves.

I would suggest to leave provision itself as a standalone repo, but let extensions have their code in a combined repo.

Here's a draft patch to generate an rc like #0 suggested.

It specifically adds '/drush' to the module path... this allows us to have code that should only run in hostmaster context in the module root.

I don't mind calling the dir provision if anyone feels strongly about it, this just seems more generic.

The advantage of keeping $options['hosting_features'] is that it's Aegir specific and not a path but a boolean.

I'm not sure exclude is being used as intended by drush ...

A list of files and directory paths to exclude from consideration when searching for drush commandfiles.", 'example-value' => '/path/dir')

My local dev has e.g. "ssl" in the exclude list, because I don't have the feature enabled. So drush would exclude any code with 'ssl' as the directory/file name. (implementation: function drush_scan_directory).

PS: now that I read it again I doubt that the '/path/dir' example is correct.

Committed, http://cgit.drupalcode.org/provision/commit/?id=a3df3facc42a90c3d0473fe8...

The first merge test for hosting_git is in https://git.drutch.nl/drupal-contrib/hosting_git

@ergonlogic: the merge method we used has a drawback ....

 git clone --branch 7.x-3.x http://git.drupal.org/sandbox/ergonlogic/2307781.git
 $ git log provision/README.txt
commit 8eadd6443a27f1c41e7c12ccaf26a16e2fef4383
Author: Christopher Gervais <chris@ergonlogic.com>
Date:   Tue Jul 22 20:21:19 2014 +0000

    Add remote_import under this project.

It only shows the single merge commit, even if you add the --follow flag.

In http://stackoverflow.com/a/20974621 I found an other way:

    $ cd hosting_remote_import 
    $ git remote add remote_import http://git.drupal.org/project/remote_import.git
    $ git fetch remote_import
    warning: no common commits
    [...]
    $ git checkout -b temp-before-merge remote_import/7.x-3.x
    $ mkdir drush
    $ git ls-tree -z --name-only HEAD | xargs -0 -I {} git mv {} drush/
    $ git commit -am"Move file to drush/ folder to prepare for subtree merge."
    $ git checkout 7.x-3.x
    $ git merge -m "Add remote_import under this project, see https://www.drupal.org/node/2300537" temp-before-merge

Please note that this currently is only about provision extensions ... not provision itself.

I've already committed the provision change in #13, so jenkins has installed with it a few times. However as the tests don't use any provision extensions ... we have no real coverage yet.

I've now pushed this for hosting_git/provision_git: #2364135: 7.x code moved to hosting_git repo

Also merged provision_logs and hosting_logs: #2410347: Aegir 3.x compatibility

Also did this for hosting_site_backup_manager in #2022809: Prepare to add this module to Aegir distribution

I see no reason to keep this issue open...