Advertising sustains the DA. Ads are hidden for members. Join today

Drupal.org Testing Policy

Last updated on
27 October 2022

In order to satisfy the variety of use-cases for testing while keeping the cost of the testing infrastructure manageable, we have established this testing policy which says what we are willing to support, and additionally what the the user experience should be for working within this framework. From a high level, this policy outlines: what code is tested, in which environments, what tests are run, when we run them, and how the end users configure their projects for these options.

Code Targets: What code do we test?

These are the various code environments that can be used to build the codebase to be tested. There are five different types of environments for core, and three for contrib, plus N number of feature branches in both. This results in a very large matrix of possible tests.

Note: Some of the described code targets are for a future state (e.g: feature branches).

Requirements

  • An issue should have the ability to define multiple code targets (i.e. an issue may be equally valid for 8.1.x as well as 8.0.x).
  • A code target is responsible for declaring its own dependencies (dependencies are not specified via a UI for testing, they are specified in .info.yml/module.info/composer.json)
  • Contrib may have multiple Major versions of a supported module at the same time.

Core

  • 8-stable: Current Core stable release - the most recent version of core that is out. : e.g. 8.0.5
  • 8-supported: Current Core patch-dev branch - Development Branch for patch development/bugfixes : e.g. 8.0.x-dev
  • 8-prerelease: Current Core minor-dev branch - Development branch for next minor version : 8.1.x-dev
  • 8-development: Next core minor-dev branch - Once the current minor-dev branch enters a beta phase, the next minor branch is opened (e.g 8.2.x)
  • 9-development: Current Core major-dev branch - Development branch for next major version : 9.0.x-dev
  • N - Feature - Current Core major-dev or minor-dev feature branch: Development branch for a major feature branch for next major release (9.0.x-bigpipe-dev)
  • Legacy core stable release: 7.29
  • Legacy core patch-dev: 7.x-dev
  • Legacy core minor-dev: 7.x-dev

Contrib

Note: These code targets assume the eventual move of contrib projects to semantic versioning.

  • Legacy contrib branch-stable (2.1)
  • Legacy contrib branch-dev (2.x-dev)
  • Contrib stable release (2.0.4)

Assuming a contrib module who’s latest release is 2.0.4, we would also see the following development branches. N because it may be possible to support more than one major branch of a contrib module at a time.

  • N - Semantic contrib patch-dev (2.0.x-dev)
  • N - Semantic contrib minor-dev(2.1.x-dev, 2.2.x-dev when 2.1.x-dev is in beta)
  • N - Semantic contrib major-dev(3.0.x-dev)
  • N - Semantic contrib minor-feature branches(?)
  • N - Semantic contrib major-feature branches(?)

When testing Contributed modules, developers should be able to select which core code target to use. There are valid use cases for testing contrib both against the latest dev, as well as the current stable release of core.

Contrib Recommendation

A general recommendation is having one commit target environment and then running the other environments as a weekly regression test, this recommendation puts less burden on the infrastructure.

Distribution Tests

Environment Targets: In which environments do we run our tests?

In principle there is a near infinite number of environments that Drupal could be tested against, but in practice the Drupal Association can only afford to support so many testing environments.

Core Development Environment

This environment is how developers prove their code works, before regression testing against other environment targets.

This is the version of PHP + Database that core development is performed against. Core and Contrib should have only one development testing environment. (No need to test against other environments until the code has been proven to work in this one).

Core Supported Environments

These environments represent the regression suite. Once code works in the development environment it should be tested against the environments that Drupal core officially supports.

e.g: PHP versions (5.5/5.6/7) + database vendors and version (mysql/maria/pgsql/sqlite) + configuration options (Innodb vs myisam, memcache vs apcu, opcache on/off etc).

Core "Targeted for Support" Environments

These are the various PHP versions + database combinations that Core does not yet fully support, but which Core is working to make supported.

An example of an environment in this category would have been pgsql + php5.5 during the summer of 2015.

Core Minimum Supported Environment

This environment is the lowest possible minimum requirements supported by Drupal. (php 5.5.9, mysql 5.5) - it is also a Core supported testing environment, but has its own label.

Ad Hoc Environments

These are environments that a test can be run in, but is not necessarily part of a regression suite.

For example, we may want to be able to run some tests against the dev branch of php7, but we wouldn't want that to be part of the regression suite.

A future state goal for the ad hoc environments is to allow user-defined ad hoc environments to be tested.

Assessment Targets: What tests do we execute?

Whenever we run tests, there are many potential tests that we want to run on those code target/environment target combinations. A grouping of tests (and any related tasks) is considered an “Assessment”. Currently DrupalCI only has one assessment defined (SimpleTest), but in future will have more defined to more accurately reflect the needs of the developers, and the context of their tests.

Note: Many of the assessment targets defined are for a future state when DrupalCI supports multiple assessment targets. In its current state, the Simpletest assessment is the only supported task within our single assessment target..

Development Assessment target

Functional tests

  • simpletest
  • phpunit
  • phantom-js mink tests
  • visual diffs
  • Behat/BDD tests

Code Quality Tests

  • code coverage
  • php codesniffer rules
  • syntax checks
  • Code style for css/js/php/twig

Regression Assessment target

Functional tests

  • simpletest
  • phpunit
  • phantom-js mink tests-

Performance tests

  • xhprof comparisons

Prerelease Assessment target

Functional tests

  • simpletest
  • phpunit
  • phantom-js mink tests

Custom Ad-Hoc Assessment target

Other test tasks that are not part of the typical workflow, but useful to occasionally run, such as:

  • Cyclomatic complexity analysis
  • Static analysis
  • etc.

Different assessment targets have different triggering events. We don’t need to run performance regressions on every patch submitted, and we probably do not need to run code style checks on contrib regression tests, for example. Code coverage reports might only need to be ran when new tests are committed.

Test Triggers: When do we run each kind of test?

These are the kinds of events that can be configured to trigger a test. Depending on the configuration some tests will be queued immediately upon triggering, whereas others will be queued on a schedule.

When a Patch is submitted to an issue

Patches submitted to issues generally need to confirm first that they work at all and that they do not break anything in the typical environment. As such, patch testing should have the following configuration:

  1. N - Code Target(s): Whichever code targets are associated with this issue. Examples:
    • 8.0.x-dev {core patch dev}
    • 8.1.x-dev {core minor dev}
    • 7.x-2.x-dev {legacy contrib}
    • 2.0.x-dev {semantic contrib patch dev}
    • etc.
  2. 1 - Environment Target: Core Development Testing Environment (currently php5.5 + mysql 5.5 - proposed php7 and mysql5.5)
  3. Assessment Target: Development assessment target

When a patch is marked RTBC. and is in the queue > 24 hrs.

When a patch is marked RTBC, we have confirmed that the patch works, and it is ready to be accepted into core or the contrib project. At this point we should now test that patch against all the other supported environments in order to ensure that we did not introduce any unanticipated regressions.

  1. N - Code Target(s): Whichever ones are associated with this issue
  2. N - Environment Targets: Whichever Environments we have defined as supported. Examples:
      • php5.6 + mysql 5.5
      • php 5.6 + sqlite,
      • php 7 + mysql 5.5
      • etc.
  3. Assessment Target: Regression Assesment Target

New commits arrive on a code target

When a new commit lands on any particular code target, we’ll want to re-test any outstanding patches that are marked RTBC in issues that are assigned that code target. Ideally we want to verify that the patches are not broken by the recent commit.

  1. N - Code Target(s): Whichever ones are associated with this issue (8.0.x-dev and 8.1.x-dev, for example)
  2. 1 - Environment Target: Core Development Testing Environment (currently php5.5 + mysql 5.5 - proposed php7 and mysql5.5)
  3. Assessment Target: Development Assesment Target

When a core committer or project maintainer is ready to commit an RTBC patch that has been re-tested. (perhaps ad hoc)

If we decide that we are only going to re-test RTBC patches with only the core dev testing environment, (see 3.a.ii above) then we will probably need to have a pre-commit sanity check on the environment to verify that the patch still works in all environments.

  1. N - Code Target(s): Whichever ones are associated with this issue (8.0.x-dev and 8.1.x-dev, for example)
  2. N - Environment Targets: Whichever Environments we have defined as supported (php5.5, 5.6, 5.7 + sqlite, postgres, mysql 5.5, mariadb etc)
  3. Assessment Target: Regression Assesment Target

When a Core “Branch” is ready to release

Core developers may, as a pre-release sanity check, wish to run a series of contrib tests, particularly for new patch level releases that establishes that they did not accidentally break any api’s that are in use by known contrib testing. Additionally, there are features of the testing environment that would differ in a production environment, like accidentally using assertions as a control structure. Having a "production mode" test environment with assertions disabled would allow a pre-release catch of these rare bugs, without incurring substantial testing cost.

  1. N - Code Targets: Any contrib code target that has a dependency upon this patch/minor/major/feature branch, and also has currently passing tests
  2. 1 - Environment Target: Core Development Testing Environment
  3. Assessment Target: Pre-Release Assesment Target

When new dependencies are released for an existing core release

When a core release happens, its composer dependencies will typically be the most recent release. As time passes, there may be new releases of those dependencies that might not be compatible with core. It would be good to test those new deps to ensure that if a contrib module overrides a core shipped dependency that we’re aware of that breakage, both for upstream and for our sake.

  1. 1 - Code Target: Current Core stable release
  2. N - Environment Targets: Core supported testing Environments
  3. Build Target: Regression Assessment Target

When an Ad Hoc/On demand test is requested. (for individual patches, or the tip of any code target).

Anywhere a test result can be displayed (issues, branches) an Ad hoc test should be able to be initiated.

  1. 1 - Code Target: Any defined code target
  2. 1 - Environment Target: Any defined environment
  3. Build Target: Any defined Build target

When an issue is reassigned to a new Code target.

When an issue gets updated to a new version, we may need to re-run tests to verify that existing patches still pass.

  1. N - Code Target(s): Whichever code targets change.
  2. 1 - Environment Target: Core Development Testing Environment (currently php5.5 + mysql 5.5 - proposed php7 and mysql5.5)
  3. Assessment Target: Development build target

This may only trigger when an issue is manually re-assigned, and not when issues are bulk updated - example of bulk update is 8.2.x is opened, existing 8.1.x issues need to move to 8.2.x, and existing 8.0.x issues need to move to 8.1.x.

Tests with External Service Dependencies

Well designed tests should be consistent and repeatable, and should avoid any dependencies on the external internet. Such external dependencies may be fragile, and result in false negatives and/or random failures due to external service outages or network issues. 

Wherever tests depend on an externally hosted resource or API, best practice should be to mock that resource or API within the test. 

For example, if a module depends on the Google Maps API, the test should mock the API request and response, rather than depending on a live request to the actual Google Maps API. After all, we are testing the Drupal code, not the availability of external services. 

Drupal Association staff will not be able to support troubleshooting of any test issues having to do with test dependencies on external resources or APIs. 

Test Priority

An additional strategy we employ to reduce the testing burden (cost, result storage, data volume, notification burden), we aggregate test triggering events into periodic testing. While it would be the most accurate to run a comprehensive suite of tests on every single triggering event, oftentimes it is sufficient to group a series of triggered events together and test them all once a day.

Priorities

  • Immediate - Tests flagged for immediate testing go into the test runner queues right away.
  • Daily - Tests flagged for daily testing wait until the next scheduled build to be submitted. These tests may accumulate events (i.e. multiple commits on a single branch during a single day for example)
  • Weekly - Tests flagged for Weekly testing might be gigantic regression suites, or slow performance tests.

Data Retention Policy

The sheer volume and scale of tests means that we cannot keep all test artifacts around forever. So we need to define what will, and will not be available after a certain amount of time.

Dispatcher build retention

Builds older than 180 days will be removed from the dispatcher, regardless of state or status.

Only the most recent passing build will be retained for branch tests (on commit, daily, weekly tests), previous failed builds in the 180 day window will be available.

Builds over 30 days old on issues that are in any sort of closed status will be removed from the dispatcher.

DrupalCI will remove simpletest_html test artifact directories after 1 week.

Drupal.org test result retention

Test result data may be deleted from Drupal.org for the following. Tests can always be re-ran if needed.

  • Issues closed for over 180 days
  • Custom branch tests ran over 30 days ago
  • Scheduled branch tests ran over 180 days ago

If test results are larger than 400M on disk, they may be deleted after 24 hours to preserve disk space on the DrupalCI dispatcher.

Help improve this page

Page status: No known problems

You can: