Original Issue

Docker has potential as a container to deploy site servers or a whole Aegir setup.

I already found one example.

Possible use cases:

  • containerizing services such as database
  • containerizing sites
  • ease installing a new aegir server

Not that I'm working on this at the moment, but just a place to organise links and thoughts....

-- helmo, July 25, 2014

Aegir on Docker

This META issue will track all the efforts going on around Docker and Aegir. There are a lot of moving parts, and a lot of different possible configurations.

Right now the goals are this, in this order:

  1. Polish and release "official" aegir hostmaster and web server Docker images. This is the "Aegir on Docker" configuration. All sites fit in the one Hostmaster container, all databases are installed on a separate database container. An officially supported docker-compose.yml and rancher-compose.yml file will be released that is designed to scale horizontally.
  2. Finalize the new automated Behat tests and improve travis integration, allowing Pull Request driven development.
  3. Submit as an official Docker image.
  4. Submit to Rancher Catalog.
  5. Work on launching docker "servers" with Aegir. I call this "Docker on Aegir" configuration. Adding database containers is relatively trivial. I think we can set it up so a new Server Node represents a new Web Server Container. Then, when verifying the web server, we can rebuild the container with the platform's volumes mapped to /var/aegir/platforms. This way, the web server containers only have access to the files selected in the front end. This would still be a relatively traditional configuration: Web and database server containers are allowed to host multiple sites.
  6. Work on PaaS Mode: Isolated clusters for each site. We've already made some progress here using docker-compose.yml files: https://github.com/opendevshop/devshop_rancher/, however, in order to continue down this route, we need to make major changes to how Aegir deploys.

Code Repositories:

Rancher is an incredible tool for managing docker systems. It has a built in REST API and CLI for launching docker clusters with built in tools for high availability, while remaining unobtrusive and simple. See https://github.com/rancher/rancher and http://rancher.com/ for more info.

CommentFileSizeAuthor
#17 Screenshot from 2017-02-15 09-11-01.png273.92 KBJon Pugh
Members fund testing for the Drupal project. Drupal Association Learn more

Comments

Jon Pugh’s picture

We are starting a project that will be exploring this.

I'm a total docker newb, but once I can get it working on my local I'm planning on diving right into this.

The way I imagine it working is that we create a new Service called "docker" that would attach to a server just like http/db. Then, we can use Server nodes to track each container. The docker containers would obviously host the other services (http, db). We should create a "docker image" select list on server nodes, so we can easily choose what docker image to use when creating the container. We would also need to create a "docker host" select list on server nodes, so you can choose which docker host to put the container on.

After much thought I think this is actually going to be easier than we think. Aegir already manages remote servers really well, so once we can wire up node/add/server to actually instantiate new docker containers, and we build an "aegir remote" docker image, this will be just about done.

After we can do that, we should be able to build web clusters (packs? Clouds? See https://www.drupal.org/sandbox/ergonlogic/2308447) out of the docker containers pretty easily as well.

helmo’s picture

I've forked the above mentioned repo and started porting it to Aegir 3.x and Ubuntu 14.04

See: https://github.com/helmo/docker-aegir-mariadb

This is aimed at having a quicker dev system ... comparable to aegir-up.

niccolox’s picture

one scenario I am imagining, which is more of an systems integration approach than a development approach is run Docker Aegir Mariadb https://github.com/helmo/docker-aegir-mariadb inside Dokku Alt

create some small incremental modules that allow provisioning / hosting control of Dokku Alt containers via ssh command line, and use Aegir multi server within Dokku Alt

if Git modules where used, it would also be possible develop sites using Devshop and have multiple remotes for repos, you could push to devshop, gitlab, dokku alt, and potentially also other versions of Dokku Alt or even Deis etc

within Dokku Alt you could create s3 storage using ceph clusters

I know this sounds complicated, but Dokku Alt is actually very simple, very powerful and quite stable

see these two threads at Dokku Alt

Using Dokku-alt with Drupal #142
https://github.com/dokku-alt/dokku-alt/issues/142

The Future of Dokku Alt
https://github.com/dokku-alt/dokku-alt/issues/139

so in summary

use Dokku Alt as a paas

run the following apps in Dokku Alt

Aegir 3 (master/hub)
Aegir 3 (remote web head)
Drupal s3 storage on Ceph cluster (minimum 2 nodes on Dokku Alt, more on other servers)
gitlab
devshop
solr

if Tinc was used, all of these apps, and Dokku Alt as a host, and external servers running more Ceph nodes, i.e. petabytes, and perhaps SolrCloud and Mariadb Galeria, could be expanded horizontally

this is a very interesting use of Tinc and Openstack
http://dachary.org/?p=2969

I guess it would be possible to even run a dedicated server or vps running Openstack hosting Dokku Alt hosting Aegir

I am currently running Proxmox at ovh running Proxmox running Dokku Alt hosting Drupal 8 / Wordpress

another possibility is create Drush extensions for Dokku Alt commands or set Dokku Alt environment variables via Aegir

anyway, I think Dokku Alt is a great Docker paas, its simple and stable and is a nice distro of Dokku, which works, has a Dokku Alt Managaer (i.e. a php based web ui) and has bundled many of the better Dokku contrib modules etc

cheers

helmo’s picture

Status: Active » Fixed

I think we're done investigating ... #2793901: Request for Contribution: Aegir on Docker

Jon Pugh’s picture

Initially I was really happy to see this old issue get marked fixed... but then I started thinking about all the work ahead.

Should we keep this one open and turn it into a detailed META issue for our Docker work?

Jon Pugh’s picture

Title: [meta] Investigate Docker.io » [meta] Aegir & Docker
Status: Fixed » Active

Yeah, now that I think about it, I've been doing so much work across so many different repos, I think it would be good for us to create a main META issue for all docker on aegir work.

Jon Pugh’s picture

Issue summary: View changes
Jon Pugh’s picture

Issue summary: View changes
Jon Pugh’s picture

Issue summary: View changes
Jon Pugh’s picture

Issue summary: View changes
Jon Pugh’s picture

Issue summary: View changes
Jon Pugh’s picture

Issue summary: View changes
Jon Pugh’s picture

Issue summary: View changes
colan’s picture

Jon Pugh’s picture

I've identified another item blocking full docker integration with Provision/hosting.

The Provision_Context_server class contains method for shell_exec, sync, and fetch.

These all work over SSH if the server is not "localhost", which is not what we want it to do if it is a Docker server.

I am very close to launching docker web containers via Aegir, but I cannot figure out a clean way to override these methods.

If you try to run a server verify on a server node with the hostname "web" or "webcontainer", the task will hang, i imagine because it's asking for a password.

The magic happens in the function provision_context_factory(). This is where we assign the class to be used. Turns out, it directly uses the "context_name" property to decide what class to use. So, I tried to override that with my new Provision_Context_server_docker class, but then the server context failed to generate, even though it extends Provision_Context_server.

If we can use this method to swap out classes when needed, then we can do things other than SSH exec'ing and SCP'ing, but doing so breaks the drush alias file.

Jon Pugh’s picture

By george, I've done it.

Screenshot of the first drupal install in aegir docker containers.

Last week I was struck by insight in how to get this to work.

  1. A new hostmaster container that includes docker client and docker-compose, setup as a privileged container so that hostmaster can talk to the docker host.
  2. A new hostmaster container that includes docker client and docker-compose, setup as a privileged container so that hostmaster can talk to the docker host.
  3. A new "Hosting Docker" module in the 'docker-services' branch of http://drupal.org/project/hosting for all new Aegir services for HTTP and DB. Likely will be extracted as contrib, but I do think its close enough to be considered for core.
  4. A new aegir/web container at https://hub.docker.com/r/aegir/web/ . This is used as the "remote" web server.
  5. docker-compose file generated for each server node. aegir services map to docker-compose services, so if you select MySQL Docker and Apache docker, both get added to the docker-compose.yml file. This will allow additional docker services to be added easily.
  6. We then add the hostmaster container to the network created by docker-compose v2 so that hostmaster can connect directly to it via mysql:3306, allowing aegir to do it's thing as a normal DB server would.
  7. A new "Hosting Docker" module in the 'docker-services' branch of http://drupal.org/project/hosting for all new Aegir services for HTTP and DB. Likely will be extracted as contrib, but I do think its close enough to be considered for core.
  8. A new aegir/web container at https://hub.docker.com/r/aegir/web/ . This is used as the "remote" web server.Volumes map /var/aegir/config/server_NAME to the container, so it only sees it's own config. Currently maps the entire platforms folder for simplicity, but we can easily add volumes to only map the platforms that are assigned to that server. (QUESTION: Does anyone know of a way to get all platforms on a server via provision? I can load from hostmaster database but the "right" way would be to use provision-only
  9. docker-compose file generated for each server node. aegir services map to docker-compose services, so if you select MySQL Docker and Apache docker, both get added to the docker-compose.yml file. This will allow additional docker services to be added easily.
  10. We then add the hostmaster container to the network created by docker-compose v2 so that hostmaster can connect directly to it via mysql:3306, allowing aegir to do it's thing as a normal DB server would.

Last task is to alter the port written to vhost template to always be 80, but it works!

fuzzy76’s picture

Would there be a point in creating an issue for each of the goals listed here? I'm trying to assess the current status of "Aegir on Docker", but I am not sure where to look.

Jon Pugh’s picture

Status: Active » Needs review

Hi fuzzy76...

It's not worth posting issues here, unless they relate to changing aegir/hostmaster/hosting/provision themselves. (or the hosting_docker drupal project). Please use the drupal issue queues for those projects.

You can see the current state of docker integration in two places:

Testing and feedback are most welcome.

fuzzy76’s picture

My question was specifically about the goals listed in the issue summary of this issue. I don't really see how that can be related to anything but hostmaster. As the first goal states "Aegir on Docker" this was the specific part of this issue I was looking into.

While the https://github.com/aegir-project/dockerfiles contains the code and commits for that part of this meta-issue, it doesn't state anywhere how functional / working / tested the code is or what it is suitable for at this stage. And since it doesn't, my natural guess would be that it is not ready for use (but I could be wrong).