I've been giving lots of thought lately to the effort involved in maintaining platforms and sites in Aegir. Unfortunately, it ends up being somewhat confusing to end users, who end up having to understand how platforms are built, that a 'migrate' task is the appropriate method to update sites, etc.

I'd like to propose the introduction of a new content type to simplify much of this: 'distributions'. A 'distribution', in this case, refers to a 'type' or 'flavour' of platform. I'm open to using another name, but this is pretty much in line with d.o's definition:

Distributions provide site features and functions for a specific type of site as a single download containing Drupal core, contributed modules, themes, and pre-defined configuration. They make it possible to quickly set up a complex, use-specific site in fewer steps than if installing and configuring elements individually.

This'd help to solve an ongoing AegirWTF, about what platforms are, and how they work.

I propose implementing an Aegir Distribution content type that would work along the following lines:

  1. When creating a distribution, the following fields would be required:
    1. name (title)
    2. git repo url
    3. path to makefile (within the repo)
    4. option to track:
      1. a branch (text field for branch name) or
      2. tags (text field for tag regex)
  2. Creating a distribution would trigger the creation of a platform, that specifies a remote makefile constructed from the 'git repo url', the 'branch' or 'tag', and the 'path to makefile' fields. In the case of a tag, it'd query the repo to find the latest tag that matches the regex.
  3. An Aegir queue would poll the repo regularly for new commits to the tracked branch, or new tags. Upon finding either of these:
    1. A new platform would be created
    2. Old platforms from this distribution would be locked.
    3. An 'update site' task would become available on all sites running on older platforms of the distribution. This would:
      1. Look up the latest platform in the distribution to which the site's platform belongs.
      2. Trigger a migration of the site to the latest platform.
    4. An 'update all sites' task would become available on all older platforms of the distribution. This would:
      1. Look up the latest platform in the distribution.
      2. Trigger a migration of all sites on the old platform to the latest one.

Comments

ergonlogic created an issue. See original summary.

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
CreditAttribution: colan at Colan Schwartz Consulting commented

I like this idea, but would makefiles be mandatory? What about straight-up Git repositories?

gboudrias’s picture

gboudrias CreditAttribution: gboudrias at Praxis Labs Coop commented

I would personally rather use a lighter version of this, in which we can simply "tag" or categorize platforms. The problem with a new content type is that you have to adjust the workflow and documentation depending on whether they have it or not, which could make support more difficult. If it's just another platform attribute, it's just a matter of saying "if this is here, add it".

And if we were to make this mandatory, I think it's simply too big of a change for Aegir3, as we would have to change all the docs.

So I'm not opposed to adding this if it's not enabled by default, but I probably wouldn't use it. Based on what you're saying though it seems like it could start as a simple contrib module, doesn't seem like we would need to remove anything.

This is just my preemptive opinion, the implementation is obviously important.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

Sure, hosting_git provides support for building platforms from git repos. It'd make [1.3] optional and alter how [2] builds a platform, but that shouldn't be too much of a challenge.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

Distributions would be optional, whether enabled or not. This is intended to simply automate the process of building new platforms, and selecting the target platform when running migrations, that's all.

gboudrias’s picture

gboudrias CreditAttribution: gboudrias at Praxis Labs Coop commented

After some further discussion on IRC, this is probably more useful than the alternative of simply tagging platforms, if we can centralize most of the logic (eg the hierarchy of which platform follows which, or at least which one is the latest).

We'd still need to adapt the contrib modules but they already don't have this, or in some cases they are implementing the logic themselves so they could be simplified.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

IRC log of said discussion: https://hefring.mig5.net/bot/log/aegir/2016-04-12#T616012

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
CreditAttribution: colan at Colan Schwartz Consulting commented

After thinking about this some more while working on #2705805: Merge SaaS suite of modules into Aegir Services, I'm definitely in favour of this. Currently, the SaaS configuration needs to be updated every time there's a platform upgrade (i.e. sites are moved to a new platform to get an upgraded codebase): a new platform ID must be specified so that new sites don't get deployed on the old one.

If we had distributions, the distribution ID would be in the config instead, and it wouldn't have to change. New sites would automatically get deployed to the distribution's latest platform.

omega8cc’s picture

omega8cc CreditAttribution: omega8cc commented

This is great idea!

Even the old idea, based on makefiles and feature servers, as explained by Vertice, also used "Distributions" name, even if it was in fact about building platforms:

This functionality will eventually evolve into the ability of an Aegir install to subscribe to new releases of distributions, like Open Atrium or OpenPublish. Coupled with Feature Servers, this will go a long way towards solving certain development workflow problems.

See for reference: "Aegir 0.4 Alpha 13: Drush Make Makefile Support Allows for Automatically Building Drupal Distributions"

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

Having thought this through further, I think it should be split into 2 projects. The first being the 'distribution' content type, along with it's "upgrade" tasks for sites and platforms. The second would be a framework for monitoring some source for changes and automating the builds of the platforms themselves.

So, following that logic, and setting aside the second compotent, a 'distribution' is pretty simple:

  1. A 'distribution' content type (no special fields needed, at this point)
  2. A node-reference field added to platforms pointing to a particular distro
  3. "Upgrade" tasks for sites and platforms, that just migrates to the latest platform in a distro.
  4. (Optional) Views of distros and platforms in a distro, related menu items, etc.

This first part is already full of wins for end-users, who will finally have a simple "upgrade" button, and alleviate their need for contextual knowledge, like which is the correct platform to upgrade to.

The second part will make the lives of platform managers easier by automating platform builds. This will be somewhat more complicated, as we'll need to support both makefile and git-based platforms; Composer too, for that matter. Also, pull- (git monitoring, lockfile diffs) vs. push-based (webhooks) mechanisms for triggering new platform builds.

However, it should provide us a convenient hook to run test suites, and eventually a full CD workflow.

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
CreditAttribution: colan at Colan Schwartz Consulting commented

Sounds good. We may want to add a type field (e.g. Drupal, Wordpress, etc.) to the Distribution content type as per #2770077: RFC: Refactor how Aegir installs Drupal., however.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented
Status: Active » Needs review

I've implemented the 'distribution' content type as per #10, over in the dev/distros branch of hosting_dev.

I borrowed heavily from hosting_migrate, fixing up how some of the comparison dialogs work, and such. Some of that could potentially move upstream.

I think it's fairly complete, for the first stage, and should be ready for testing.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

A few design decision explanations:

  1. I used Fields, Features and Context to build out the "Distribution" content type, views, block placement, etc. This greatly simplified things, since I didn't have to implement a db schema, and all it's related hooks, etc.
  2. There is no backend, nor tasks that run on Distributions. It does add "Upgrade" tasks to sites and platforms, but these simply add migrate tasks to the queue.
  3. The Distribution content type currently only has a single field: "upgrade target". This is an entityreference pointing towards one of the platforms that are a part of the platform. Furthermore, I added a "Distribution" entityreference field to platforms.
  4. Permissions are pretty broad: "create [site|platform] upgrade task". Later, we might want to add field-level permissions to control access to specifying a platform's distribution, and a distribution's upgrade target, for example.
colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
CreditAttribution: colan at Colan Schwartz Consulting commented

So to prepare for an upgrade, you would:

  1. Create a new platform pointing to the distribution.
  2. Edit the distribution's upgrade target to point to the new platform.

Is this the intended process? I suppose this is safer than assuming the newest platform using the distro, but requires manual work.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

Yes, that's correct. I wanted to keep distros relatively minimalist, for the moment, to be expanded upon later with optional extensions.

The point of this first phase is to make the upgrade process as simple as possible, from an end-user/site-owner standpoint. The second phase will deal with automating the platform maintenance aspects. I figure that'll be a good opportunity to integrate Composer support, for example. Setting the upgrade target to the latest platform (among other things) should be pretty easy, at that point.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

I've added #2864854: [meta] Enhance distributions to discuss future directions.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

I've split Distributions into a separate module: https://www.drupal.org/project/hosting_distribution. I've also implemented the Drush makefile tracking as per #2864854: [meta] Enhance distributions. I'll be releasing a beta shortly, to encourage testing.