Drupalci Roadmap

Phase I: #2949209: Phase I: Allow drupalci.yml to influence the build

Phase II:

I've done some analysis and architecting and have come up with a plan that will address several things we have on the radar:

Some major goals that we want to address with the drupal ci service are:
1. Dispatcher is an open jenkins. This has been shown, time and time again to be a security hazard, and also requires maintenance and upkeep of the OS and software. We would like to replace that component with something simpler/safer.
2. Container management is 100% on the maintainers of drupalci, which is really just me. It would be nice to expand that to allow developers the ability to manage any container setup that they need. (the original goal of 'phase II' - allow developers to submit custom environments.
3. Now that we have self hosted gitlab, we should be looking at how it can solve some of the things that Drupalci is currently doing, and remove maintenance burden while simultaneously enabling features for developers.

As a brief overview, drupalci is made up of the following four components:

1. project_issue_file_test
2. dispatcher.drupalci.org
3. drupalci_testbot
4. drupalci_environments

Their high level roles are:

• Configure project testing (pift)
• Trigger builds (pift)
• Summary of build results and issue status interactions (pift)
• Orchestration of aws "Testbots" (dispatcher)
• Artifact and detailed results storage (dispatcher)
• Manage php, database, chromedriver containers. (drupalci_environments)
• Manage the Host AMI, and its corresponding Vagrant Box (drupalci_environments)
• Environment establishment on a testbot (drupalci_testbot)
• Codebase build (drupalci_testbot)
• Running assessments (drupalci_testbot)
• Gathering build artifacts and making them available to the dispatcher (drupalci_testbot)

In order to address the major goals above, Im proposing that we transition to using Kubernetes as the component that is responsible for both

• Orchestration of aws "Testbots"
• Environment establishment on a testbot
• Manage the Host AMI, and its corresponding Vagrant Box

And transition to gitlab's CI/CD for

• Trigger builds (pift)

In order to accomplish this, it will be a multi phased approach of this phase, so yeah, now were swimming around in subphases.

Subphase A: Kubernetes

The first step that is required, is to take the work of building the Codebase out of the 'drupalci host'. We can accomplish this by creating another 'build' container such that the only thing that is happening on the current testrunner outside of a container would be the environment establishment. All of the rest of the commands should happen inside of the container environment. This will have the added advantage of reducing the amount of path mappings back and forth between the containerized view of the filesystem and the hosts view of the filesystem. (this includes other commands running directly on the host, such as eslint etc)
So, Build the container first:
#3156947: Create a new buildcontainer
Then use it:
#3156948: Utilize build container to create the codebase

The second step is to split up the testrunner source such that the parts of the source responsible for envirionment establishment, and the parts that actually run inside the established environment are separate. The goal here is to get to a place where we can create the environment any number of ways (using docker compose, using kubernetes, using drupalci_testbot on an ami/vagrant box. and have that be separate from "executing the build". This means that a *lot* of the build plugins will have to assume they are running from within a build container, and are executing commands on the other containers within the integrated container environment. (i.e. the build will no longer be 'running' on the host, and the `execEnvironmentCommands` function on buildTaskBase should assume intracontainer process communication)
Essentially this means replacing most instances of 'execCommand' with something that knows how to do intracontainer work. The assumption is that the 'build' will be running on the buildcontainer, and any given buildStep will need to execute on the php container. We need to find a way to do that. It might be sshd running on all the containers, or it might be that at this point, we abandon vagrant/docker etc, and rely on the kubernetes API for intracontainer communication. TBD.

***TODO OPEN A DRUPALCI_TESTBOT ISSUE FOR THIS***

Third, we'll need environments to be first class citizens on drupal.org. Right now they are just a big ol nested array on drupal.org, but we need them to be richer entities so that we can work with them properly. Once we have them available to send to drupalci, they can be serialized as either a docker compose.yml file or as a kubernetes pod definition, and sent to drupalci.

***TODO OPEN A PIFT ISSUE FOR THIS***

Once we have the environment separated from the build, we can establish a kubernetes cluster on aws, backed with spot requests (https://aws.amazon.com/blogs/compute/cost-optimization-and-resilience-ek...). We can then replace jenkins's EC2_fleet plugin with the jenkins kubernetes plugin: https://www.infoq.com/articles/scaling-docker-with-kubernetes/ and https://plugins.jenkins.io/kubernetes/)

***TODO OPEN A DRUPALCI ISSUE FOR THIS***

At this point we will have the first subphase of drupalci complete: Builds running in a k8s environment, with local drupalci using minikube instead of vagrant.

Subphase B: Kill Jenkins

This is where we transition from using jenkins to using gitlabci/cd backed by kubernetes.

It will require that all of our PIFT integrations are also able to trigger jobs on gitlabCI/CD via the api, because we do not want to force everybody into a merge request workflow until they are ready for it, so some patch testing will still be needed. (maybe by that time MR's can be the one true way.. who knows). In any case we'll have to evaluate gitlabs triggering mechanisms and configuration tools to ensure that we dont lose functionality when transitioning from pift -> gitlab.