Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.I've been working hard to create a scalable Aegir cluster using docker-compose.yml.
For those of you that don't know, docker-compose allows you to specify "scale" of a service, meaning you send the command docker-compose scale hostmaster 2 and docker will automatically create a new container with all the links needed to scale...
Except it won't work, because Aegir explicitly grants permissions to the site's databases only to requests coming fro the site's chosen Web Servers.
So when we "Scale", meaning adding new web server containers, it fails, because those new web server containers are not allowed to access the specific database created for that site.
I've tracked this down to Provision project: /db/Provision/Service/db.php
By applying this patch, the database is accessible from any host using the generated username, password, and database name.
- foreach ($this->grant_host_list() as $db_grant_host) {
+ $hosts = $this->grant_host_list();
+ $hosts[] = '%';
+ foreach ($hosts as $db_grant_host) {
This is a total hack but it did allow me to get access.
Proposed solutions:
- Create a new Provision_Service_db_docker class with a replacement for grant_host_list() method. This seems like overkill and feels confusing, especially when thinking about how to use that on hostmaster install.
- Create a new property on site context: "db_grant_hosts". Instead of dynamically generating the hosts list in the Provision_Service_db class, we can write the list to the site context and then just read it back. This would allow the front-end to alter the list of db_grant_hosts using a simple node alter. Then, we can create a hosting_docker module with a service that inherits from Hosting_Service_mysql, and simply alters the db_grant_hosts property.
- ??
This one needs some thinking. As we start to use Docker/Kubernetes, we no longer need to set access controls in MySQL itself, but should control access via docker networking.
| Comment | File | Size | Author |
|---|---|---|---|
| #16 | 2794915-grant-all-hosts.patch | 4 KB | Jon Pugh |
| #12 | 2794915-grant-all-hosts.patch | 2.21 KB | Jon Pugh |
Comments
Comment #2
ergonlogicThe database service already supports scaling web servers.
Provision_Service_db::grant_host_list()doesn't return static values, but rather requests a list of servers from the HTTP service, thus allowing it to scale across web clusters and packs. So you might want to investigate adding something like addingProvision_Service_http_docker::grant_server_list()instead.I think Provision_Service_db::grant() only gets called when creating the site database; so during install/migrate/clone tasks. We might want to consider also calling it during "verify" tasks, or the like, to allow us to pick up new hosts added to clusters and such.
Think of MySQL as using 2-factor authentication, checking both the user's hostname and password. Moving to a simple wildcard (%) would essentially reduce security to just the password. However, wildcards aren't all-or-nothing. We could, for example, open access to others hosts on the same subnet using:
192.168.10.%or192.168.10.0/255.255.255.0(ref.: MySQL Access Control docs).Comment #3
Jon PughI'm not saying we should change the default behavior, I'm just pointing out we need an easier way to alter this.
I was actually looking at the Cluster/Pack classes as well when trying to figure out how I might be able to lead by their example, but in the end I realized this isn't something Aegir should even be worrying about.
What I am saying is that as we start using docker, we don't have to worry about the grant_host_list() thing at all. All of the access control between database and web servers can be handled "upstream" at the docker orchestration level.
Docker compose v2 creates a wholly separate network for each cluster. When using database containers, you typically don't worry about host restrictions in MySQL at all, as the restrictions are better handled by docker networking itself.
What I am looking for is a decent way for me to start getting this working now, without major changes to Provision.
I'll keep digging. I've updated the title and summary to be more specific.
Comment #4
Jon PughThis is the part I'm trying to avoid. Docker compose let's me scale up the web containers by number. Docker compose maintains a network layer so they already have access to the DB.
I'd much rather let docker links/networking decide who can access the database container then worry about aegir and MySQL itself keeping track. I have a feeling most users of Docker database containers no longer GRANT access by host?
Comment #5
Jon PughComment #6
ergonlogicCan you point to where you're running docker and/or docker-compose from Aegir?
Comment #7
Jon PughSorry for the delay. I've got 2 really good excuses, at least.
https://www.drupal.org/project/hosting_docker
Comment #8
helmo CreditAttribution: helmo at Initfour websolutions commentedComment #9
JamesK CreditAttribution: JamesK at Advisor Websites commentedI'm using web pack to manage a server cluster but I run into lots of issues with this stuff as well. My database is on AWS Aurora and only accessible on a private network, so the host-based access control is a huge frustration to me as well because every time the cluster changes I have to manually update the cluster config in Aegir and manually update the mysql.user/.db tables with a new user hostmasks.
Removing host restrictions would be a good step towards real cluster support.
Comment #10
Jon PughJamesK see the patch in https://www.drupal.org/node/1392934.
That should actually work for you for the time being.
Comment #12
Jon PughWell, I was inspired.
The new branch and patch do the following:
Comment #13
Jon PughI made the changes this way so it's the least disruptive to default aegir behavior: If the value is set to false or not there, it will do the usual.
Users must set the new property manually in the ~/.drush/server_NAME.alias.drushrc.php file.
Comment #14
Jon PughNow I just need a way to set that in hostmaster-install, and we'll be good to go on Docker!
Comment #16
Jon PughDone.
hostmaster-install now has an option --aegir_db_grant_all_hosts that passes to the created database server context.
Updated patch attached.
Comment #18
helmo CreditAttribution: helmo at Initfour websolutions commentedCommitted, with only a false => FALSE style change.