Contributing to DrupalCI

Last updated on

25 January 2017

Contribution Guidelines for DrupalCI

DrupalCI is critical dependency of the development process for Drupal. All contributions to core, and any contributions to contributed modules that have enabled automated testing will be tested through the DrupalCI system. As such, it is critical that DrupalCI code is production-stable and that it is reviewed by DA staff to avoid disruption to Drupal development.

Three principles of contribution to DrupalCI

DrupalCI's implementation on Drupal.org is a production service, serving thousands of users on Drupal.org, and supporting many thousands more by testing the code that goes into Drupal. Stability is paramount.
DA staff must understand and approve changes to the DrupalCI code base. Because DrupalCI is supported as a production service on Drupal.org, Association staff are ultimately responsible for its stability. This is not intended as a bottleneck to contribution, but rather as a safeguard to ensure availability of a stable production service.
DA staff time for code review and approval of contributed code is prioritized according to our internal roadmap, and to meet the needs of Drupal core development. Code contributions that are not aligned with this roadmap may not receive prompt review.

The current DA staff roadmap for DrupalCI

Updated: 2016-03-30

Hot fixes required to support Core development.
Code that enables testing
Functional tests that exercise the system
Refactored code + unit tests for the new code that decouple and simplify the system.
Functional tests for changes to existing behavior + changes + unit tests
Functional tests for new features + new feature code + unit tests for the new features
Maintenance and upkeep
New features (with functional tests + unit tests.)

Test Coverage for DrupalCI

Functional Tests

We want to have end to end functional tests such that a given configuration of DrupalCI will exercise a known, common path through the system. The function tests should be built such that they verify that a particular system or subsystem reacts the way that we expect it to, and reacts the way we currently *want* it to.

The functional tests in this situation are combination of local configuration, environment variables, and the command line arguments.

Functional tests will allow us to use the exact same set of standards to say that the code still does what it needs to, without the risks of manual testing methods.

Additionally, having functional tests will allow us to ensure that we have a more comprehensive regression suite that will alert us to any problems with the code before it gets deployed into production. Functional tests are the coarsest level of quality assurance, and the first line of defense for maintaining service stability without regressions.

All functional tests will be reusable as they will all be essentially black box tests, and should only change if we are changing the external api - i.e. if we add a feature that provides new output, we add a test that proves that the new output is changed. If we change or combine how environment variables are handled, we’ll need to change how our tests handle those variables.

Strategy for implementing functional tests

Firstly we’ll follow some patterns established here:
https://www.drupal.org/node/2676854

We will start with basic testing of the commands. Most of them do not require a super robust suite of inputs, as the only ones that have production impact are the ‘run’ command and the ‘docker-rm’ commands. The goal is to have enough configurations that exercise not only the “does this work” path, but additionally the standard paths through the system (things like sqlite as the db - no db container, or a contrib file with no patch, and a d7 test - essentially each “normal” scenario we can come up with, plus ensuring some typical error states, like invalid states etc.)

Once we have a basic set of function tests that pretty much test the spec of the program as it currently stands, we can move on to more targeted tests that may exercise specific pieces internally, given a set of known inputs. For example, to make sure that swapping out docker libraries doesnt have any impact, we’ll want functional tests that lead us through every path that interacts with docker, including some error states.

Unit Test

Unit tests will give us a more fine grained proof of quality, and additionally reveal elements of the codebase that are tightly coupled, as they tend to be difficult to test as a result of all of the interdependencies on other parts of the code.

Unit tests allow us to break up parts of the subsystems and ensure that they work independent of the rest of the system, giving us the flexibility to change the system in the future in response to future needs.

In almost all cases we will not create unit tests for code we know that we’re planning on gutting or refactoring, as much of that refactoring work is what will make it testable in the first place. We will aim to *add* unit tests as we refactor. Unit testing strategy should be such that we write tests as we refactor. (Ideally *before* we refactor, with broken tests, and the end result is passing tests proving our code works, i.e TDD.)

Components of DrupalCI you can contribute to

Containers

The containers can be found in the /containers subdirectory of the DrupalCI: Test Runner project. Contributors should use the production branch, which contains the latest stable code.

The Containers are published on DockerHub. The Canonical source for the code is the Drupal.org project. However, since docker hub is hardcoded to a BitBucket repository, it pulls from the BitBucket mirror.

Containers rebuild once a day in the following order:

base -> php-base -> web-base -> web--
base -> db-base -> dbname

This process takes a couple of hours. You can see where things are currently at with a given container by clicking its Build Details tab, e.g. https://registry.hub.docker.com/u/drupalci/db-mysql-5.5/builds_history/171387/

If it says Finished you are good. If it says Building it's still in process.