Support drupal being in a subdirectory of a git repo

Right now hosting_git assumes that the platform path will be the root of the git repo as well. We definitely want the platform path to remain the Drupal root, since everything else in Aegir assumes that's what "platform path" means. So we'll need to add some way to specify the git repo path.

This could default to the platform path, but be configurable to allow a parent directory to be specified. Alternatively, we could add a field to specify the path to the Drupal within the repo, but that seems like it'd be more complicated.

Either way, we'd probably want to validate that the git path is a parent of the platform path.

It looks like the back-end already supports this: http://cgit.drupalcode.org/hosting_git/tree/drush/provision_git.drush.in...

So we should just need to pass the "repo_path" parameter into the platform context/alias.

We also really need this for a project we are working on. I spoke with Jon about it and he mentioned he would also help.

Branch pushed and patch attached for review.

I tried to figure out many different ways to go about this. It proved too difficult to just store a relative docroot path because it didn't feel right to dynamically alter the Platform Path.

Instead I added a field "repo_path" which is a full absolute path to the repository. I then added the best help text I could think of to describe how to use this:

Defaults to the Publish Path if left blank. If Drupal is not in the root of your repository, enter the absolute path to clone the repository to, and edit the Publish Path to point to the correct subfolder. For example, if your Drupal code is in docroot, set Repository path to /var/aegir/platforms/NAME and Publish path to /var/aegir/platforms/NAME/docroot

It works, maybe the UI could be cleaner, but not without some javascript.

So far so good!

I'm now using this patch in devshop and it's working well.

Looks good on first glance... but the backend code? Devshop used other code for that?

I added commits to fix the version number of the update hook, fix the needle/haystack order in the validate hook and some whitespace.

I created a local test repo with:

  mkdir testrepo_d7_in_subdir
  cd testrepo_d7_in_subdir/
  git init
  drush dl drupal-7
  git add -A 
  git ci -m 'core 7.53 in a sub folder'

After creating a platform I get in my log:

Running git clone --recursive '/tmp/testrepo_d7_in_subdir' '/var/aegir/platforms/test-sub/drupal-7.53' --branch 'master'

So I think the existing repo_path option was designed for something else.

DevShop force clones repos into /var/aegir/projects/$PROJECT/$ENVIRONMENT (There's a setting for the base path.)

Then we store "path to drupal" for each project. When "environments" are created, a platform node is created and the publish_path property is generated by the pattern above + path_to_drupal.

The repo_path was meant as the location to clone the repo, if it were different from the platform's publish path, I'm not sure if it always respects that.

Aside: Thanks to this patch, I was able to get Aegir to build a platform from a makefile inside the git repo by removing the check that prevents git and make from both being used, and by patching provision slightly to prevent drushrc.php file from being generated post-provision-git-clone: http://cgit.drupalcode.org/provision/commit/?id=c50a0c0

If you set Publish Path to the location you want drush make to build the site, and set the Repository path to a separate path, and set the path to the makefile in the Repository path, Aegir will clone the repo, then run drush make on the makefile in your repo, building it in the platform publish path.

After that, I realized it might be nice to have totally separate repo_path and publish_path, so I removed the check for enforcing the platform path is inside the repo path. Then I was able to set a repo path of /var/aegir/repos/NAME and a publish path of /var/aegir/platforms/NAME and it worked!

THEN I started thinking about what to do on "git pull" for platforms built from makefile, and things got problematic:

1. We can't use drush make to "rebuild" the existing platform, as drush make won't overwrite an existing folder. (Also this doesn't follow "strict operational procedures" of aegir by using migrate tasks to update.)
2. We could use drush make to build the stack in a temporary folder, then copy all of the files over the platform (after deleting platform files without deleting the sites folder :( ) but this too wouldn't follow SoP, but it would be acting more like traditional git pull deployment..
3. We could use drush make to build a new platform, but we would need to dynamically determine the platform name and path if we want it to update on 'git push', and then it would need to attempt to migrate all sites into the new platform, and then we are left with a platform with a git URL and a second platform with just a drush make file, so we would need something to keep them "linked" so we know what platform to migrate from and to.
4. Then, this whole structure gets confusing and falls apart. I am starting to think we need a separate "repository" entity to do this properly.

We need a better way to track platforms that are upgrades of other platforms. I think we should look at Blue/Green deployment strategy and design something around that: Aegir platforms shouldn't live on their own. Each site should have a blue platform and a green platform, and the sites get migrated between the two. After a migrate, the opposite platform can be destroyed and rebuild with the next version.

The work I've done on this patch and in devshop over the years has made it clear we need a separate object to store the git repo URL.

This is because when using blue/green (aegir migrate) based deployments, the old platforms get destroyed. If we destroy the platform we destroy the git url record in the database.

I'm going to keep working on this in devshop. We store the git URL in project nodes so we should be able to create a destroy platforms before we come up with a solution in hosting_git.

Platforms are not automatically deleted when a site is migrated off of it, in vanilla Aegir. That must be DevShop behaviour.

I didn't say it was automatic.

If a platform is hosting sites, then you make a new platform and migrate sites to it, the old platform becomes obsolete and (typically) would be (should be) deleted eventually.

The work I've done on this patch and in devshop over the years has made it clear we need a separate object to store the git repo URL.

This would get solved by #2704799: RFC: Add a 'distribution' content type, wouldn't it?

Maybe we could use these in Aegir 3 without them being necessary (so as not to break the API).

Almost solves it, Colan, but not quite! :D

Think of "Distribution" as the open source part. Typically it has releases, thoroughly vetted code, and it's packaged up and distributed on drupal.org. Sometimes it's a custom distro.

Every site start off as a clean distro (core is a distro) but they quickly become unique. A website's codebase should be able to grow in it's own unique way, leading us to want to put each site in it's own git repo.

In order to start this phase of uniqueness, you want to get an exact copy of your site running, then modify the codebase slightly to see the effects, rinse, and repeat.

So I think the best solution is a "Project" node. This is not only a logical grouping of sites and platforms, but think of it as a way to save default values for new platforms and sites created within the project.

The project stores the git URL, a unique project name, a base path for all platforms in the project, and things like "path to docroot" and "path to makefile", so that you can create a new site AND platform in one form, only asking the user for an "environment" name and a git reference.
The platform path, domain name, and git repo URL are all set automatically for the user.

This is what makes devshop appealing for developer workflows. It's much less clicking and much less thinking to get sites up and running within the context of a "project". You don't have to decide a unique platform path or URL pattern every time, it's automated.

The main problem using the "platform" as the object that the git repo is stored against is that you want to build a new platform when there is new code to "pull". So in devshop, we are running "git fetch" first to detect new code without changing the current files, then analyze the output to determine we need to create a new platform. Creating a new platform automatically (without any input) from an existing platform is very difficult because we don't have any idea of where to put it or what to name it.

Having a separate "project" node solves that problem.

What else is needed to get this one in?

Pushed a few more commits. It's looking good to go. We could verify all of this by adding to the aegir behat tests.

New patch attached, just a diff of 2838489-repo-platform-path

It seems like we could simplify the UI by hiding the "Repository path", and adding a "Docroot" (or something) field. This field would represent a relative path, within the git repo. In our submit handler, we could then copy the "publish_path" field to the "repo path", then tack on our "docroot" field to "publish_path".

It'd probably also be worthwhile to add back the validate handler, and ensure that the publish path is within the repo path (unless they're identical, of course.)

I've already tried to do that, and went through many different attempts at this UX.

"Publish Path" must always be the full path to Drupal, short of re-engineering all of Aegir.

When you introduce relative docroot you end up having to mash strings together and break them apart so you can store the publish path, it gets very messy.

What really needs to happen is to totally redesign this form, which should change depending on how the user wants to build the platform.

After much research this is how we must store the data given aegir's current setup, so I suggest we commit this and worry about UI in another issue.

As for validation, I removed it prematurely when working on the makefile-in-repo feature. In that situation, the publish path doesn't have to be in the repo.

What I would really like to do is to deprecate Hosting Git altogether, and add native Git support to hosting_platform, handled just like "makefile". It would greatly reduce the complexity and maintenance of all of this.

The attached patch implements the solution I suggested in #26.

I added one commit to hide the docroot value when it's empty

@ergonlogic I also added your last patch to the feature branch

I think we can merge this ....

Uploading a patch file for easier review.

Merged!