I've been working hard to create a scalable Aegir cluster using docker-compose.yml.

For those of you that don't know, docker-compose allows you to specify "scale" of a service, meaning you send the command docker-compose scale hostmaster 2 and docker will automatically create a new container with all the links needed to scale...

Except it won't work, because Aegir explicitly grants permissions to the site's databases only to requests coming fro the site's chosen Web Servers.

So when we "Scale", meaning adding new web server containers, it fails, because those new web server containers are not allowed to access the specific database created for that site.

I've tracked this down to Provision project: /db/Provision/Service/db.php

By applying this patch, the database is accessible from any host using the generated username, password, and database name.

-    foreach ($this->grant_host_list() as $db_grant_host) {
+    $hosts = $this->grant_host_list();
+    $hosts[]  = '%';
+    foreach ($hosts as $db_grant_host) {

This is a total hack but it did allow me to get access.

Proposed solutions:

  1. Create a new Provision_Service_db_docker class with a replacement for grant_host_list() method. This seems like overkill and feels confusing, especially when thinking about how to use that on hostmaster install.
  2. Create a new property on site context: "db_grant_hosts". Instead of dynamically generating the hosts list in the Provision_Service_db class, we can write the list to the site context and then just read it back. This would allow the front-end to alter the list of db_grant_hosts using a simple node alter. Then, we can create a hosting_docker module with a service that inherits from Hosting_Service_mysql, and simply alters the db_grant_hosts property.
  3. ??

This one needs some thinking. As we start to use Docker/Kubernetes, we no longer need to set access controls in MySQL itself, but should control access via docker networking.


Jon Pugh created an issue. See original summary.

ergonlogic’s picture

The database service already supports scaling web servers. Provision_Service_db::grant_host_list() doesn't return static values, but rather requests a list of servers from the HTTP service, thus allowing it to scale across web clusters and packs. So you might want to investigate adding something like adding Provision_Service_http_docker::grant_server_list() instead.

I think Provision_Service_db::grant() only gets called when creating the site database; so during install/migrate/clone tasks. We might want to consider also calling it during "verify" tasks, or the like, to allow us to pick up new hosts added to clusters and such.

Think of MySQL as using 2-factor authentication, checking both the user's hostname and password. Moving to a simple wildcard (%) would essentially reduce security to just the password. However, wildcards aren't all-or-nothing. We could, for example, open access to others hosts on the same subnet using: 192.168.10.% or (ref.: MySQL Access Control docs).

Jon Pugh’s picture

Title: Refactor database permissions granting to allow scalable web servers. » Refactor database permissions granting to allow scalability to arbitrary web servers.
Issue summary: View changes

I'm not saying we should change the default behavior, I'm just pointing out we need an easier way to alter this.

I was actually looking at the Cluster/Pack classes as well when trying to figure out how I might be able to lead by their example, but in the end I realized this isn't something Aegir should even be worrying about.

What I am saying is that as we start using docker, we don't have to worry about the grant_host_list() thing at all. All of the access control between database and web servers can be handled "upstream" at the docker orchestration level.

Docker compose v2 creates a wholly separate network for each cluster. When using database containers, you typically don't worry about host restrictions in MySQL at all, as the restrictions are better handled by docker networking itself.

What I am looking for is a decent way for me to start getting this working now, without major changes to Provision.

I'll keep digging. I've updated the title and summary to be more specific.

Jon Pugh’s picture

So you might want to investigate adding something like adding Provision_Service_http_docker::grant_server_list()

This is the part I'm trying to avoid. Docker compose let's me scale up the web containers by number. Docker compose maintains a network layer so they already have access to the DB.

I'd much rather let docker links/networking decide who can access the database container then worry about aegir and MySQL itself keeping track. I have a feeling most users of Docker database containers no longer GRANT access by host?

Jon Pugh’s picture

Issue summary: View changes
ergonlogic’s picture

Can you point to where you're running docker and/or docker-compose from Aegir?

Jon Pugh’s picture

Sorry for the delay. I've got 2 really good excuses, at least.


helmo’s picture