Customizing DrupalCI Testing for Projects

When testing a project on drupal.org, DrupalCI will run a predefined set of tasks that:

• establishes a testing environment (php/databases/etc)
• constructs a testable codebase
• runs validations and assessments (coding standards and tests).

Oftentimes a project may need to perform additional tasks that are not part of the predefined build process or may wish to configure how the build responds to certain types of failures conditions (some maintainers may wish for coding standards failures to immediately stop the build, while others may wish for the build to proceed).

This guide is intended to inform Drupal core and contributed project maintainers how they can customize DrupalCI to suit their testing needs.

The Build Definition

The DrupalCI testbot uses a YAML-formatted file to specify a 'build.' The build definition file specifies which tasks to run, in what order, and which configuration options to use for how those tasks should behave.

Build Definition Organization

The build definition file consists of a hierarchical list of build tasks to execute, along with their configuration values. The tasks are nested in major stages, minor phases, and individual tasks.

If you look at the default build definition file, you'll see that it has a top-level key of build.

Under the build level we have the three major stages:

• environment
  • The environment stage is responsible for creating the environment that the test will run in, and start all of the services needed by the test, such as apache/php/database/chromedriver/phantomjs
• codebase
  • The codebase stage is responsible for constructing a preliminary testable codebase for the project in question and involves git cloning/composer/patching and anything else
• assessment
  • The assessment stage will run any codebase validation checks like coding standards and linting, and also executes any actual testing frameworks, like SimpleTest, PHPUnit, and Nightwatch.

Within these stages are 'phases,' such as validate_codebase and testing, which allow for logical grouping of child 'steps', 'stages', 'phases', and 'steps' are all considered "Tasks" and correspond directly to a plugin that runs in the DrupalCI codebase.

Some keys have "Labels", indicated by a period in the key, like the run_tests.standard and the run_tests.js. If the same plugin needs to be run more than once in a build, it must have a label. This will cause DrupalCI to run the same step twice but with different configuration values each time. The label is arbitrary but required if you need the same plugin to run more than once. Execution artifacts associated with that plugin will be stored in a directory with the label on Jenkins.

Some of those keys correspond to the annotated tag of a plugin that will be executed, and some keys are considered configuration for the parent plugin. As a general rule of thumb, plugin names have underscores in them, and configuration values will have dashes (unless they are just one word, in which case they're ambiguous)

Overriding Default Behavior

DrupalCI provides a mechanism for maintainers to completely replace the default assessment stage of the build, with their own configuration by placing a drupalci.yml file at the root of their project.

The drupalci.yml will look like a build.yml file, but with the codebase and environment phases omitted and having only the assessment phase. Drupal core ships with a drupalci.yml file that we can use as an example starting point.

See git.drupalcode.org/project/drupal/blob/HEAD/core/drupalci.yml:

# This is the DrupalCI testbot build file for Drupal core.
# Learn to make one for your own drupal.org project:
# https://www.drupal.org/drupalorg/docs/drupal-ci/customizing-drupalci-testing
build:
  assessment:
    validate_codebase:
      phplint:
      csslint:
        halt-on-fail: false
      eslint:
        # A test must pass eslinting standards check in order to continue processing.
        halt-on-fail: true
      phpcs:
        # phpcs will use core's specified version of Coder.
        sniff-all-files: false
        halt-on-fail: false
    testing:
      # run_tests task is executed several times in order of performance speeds.
      # halt-on-fail can be set on the run_tests tasks in order to fail fast.
      # suppress-deprecations is false in order to be alerted to usages of
      # deprecated code.
      run_tests.phpunit:
        types: 'PHPUnit-Unit'
        testgroups: '--all'
        suppress-deprecations: false
        halt-on-fail: false
      run_tests.kernel:
        types: 'PHPUnit-Kernel'
        testgroups: '--all'
        suppress-deprecations: false
        halt-on-fail: false
      run_tests.simpletest:
         types: 'Simpletest'
         testgroups: '--all'
         suppress-deprecations: false
         halt-on-fail: false
      run_tests.functional:
        types: 'PHPUnit-Functional'
        testgroups: '--all'
        suppress-deprecations: false
        halt-on-fail: false
      # Functional JavaScript tests require a concurrency of 1 because there is
      # only one instance of PhantomJS on the testbot machine.
      run_tests.javascript:
        concurrency: 1
        types: 'PHPUnit-FunctionalJavascript'
        testgroups: '--all'
        suppress-deprecations: false
        halt-on-fail: false
      # Run nightwatch testing.
      # @see https://www.drupal.org/project/drupal/issues/2869825
      nightwatchjs:

build:
  assessment:
    validate_codebase:
      phplint:
      csslint:
        halt-on-fail: false
      eslint:
        halt-on-fail: true
      phpcs:
        sniff-all-files: false
        halt-on-fail: false
    testing:
      # container command below updates phpunit as an additional build step for core, for example.
      container_command : 
        commands: "cd ${SOURCE_DIR} && sudo -u www-data composer update phpunit/phpunit --with-dependencies"
      run_tests.phpunit:
        types: 'PHPUnit-Unit'
        testgroups: '--all'
        suppress-deprecations: false
        halt-on-fail: false
      run_tests.kernel:
        types: 'PHPUnit-Kernel'
        testgroups: '--all'
        suppress-deprecations: false
        halt-on-fail: false
      run_tests.simpletest:
         types: 'Simpletest'
         testgroups: '--all'
         suppress-deprecations: false
         halt-on-fail: false
      run_tests.functional:
        types: 'PHPUnit-Functional'
        testgroups: '--all'
        suppress-deprecations: false
        halt-on-fail: false
      run_tests.javascript:
        concurrency: 1
        types: 'PHPUnit-FunctionalJavascript'
        testgroups: '--all'
        suppress-deprecations: false
        halt-on-fail: false
      nightwatchjs:

Almost any container_command you run should probably be prefixed with 'sudo -u www-data' so as to not get into permissions issues.

Adding Additional Containers

In some instances the project you are testing requires additional services running for your tests to function.  You can add containers to the build by using the host_command like the following example from the https://www.drupal.org/project/ics_field project:

      host_command:
        commands:
          - "docker run -d --name icalvalidator --rm stefanospetrakis/icalvalidator2http"
          - "docker network connect --alias icalvalidator drupalci_nw icalvalidator"
        halt-on-fail: true

This requires that the containers be published somewhere like docker hub where they can be found by name.
N.B.: Setting halt-on-fail to false may help alleviate issues with these commands blocking the whole build, even though the container(s) operation starts normally.

For further details on how this works, please see: https://www.drupal.org/drupalorg/docs/drupal-ci/custom-docker-images-in-...

Disabling Plugins

Sometimes certain build steps are not relevant to a particular project, and can thus be disabled. For example, a backend module with no front end might disable eslint and csslint validation steps, and disable the FunctionalJavascript testing run_tests build step.

*currently* the way to disable a build step is to comment it out (preferred), or delete it entirely (less desirable). There is a proposal (https://www.drupal.org/project/drupalci_testbot/issues/2975095) to add a configuration option disabled: true to every build step to allow maintainers to be explicit about which tasks are disabled. This will also allow DrupalCI to introduce new build steps and have them execute without maintainer intervention, because in the future we may introduce new build task plugins for things like code coverage, stylelint (for CSS), and additional build types (regression vs development vs static analysis).

Further Examples

Every build that executes on DrupalCI will create a fully configured, reproducible build.yml that can be found by navigating to any test result on drupal.org, clicking on 'view results on dispatcher' and clicking on the build artifacts link.

Example

Click on any test result in the issue queue.

Select the View results on the dispatcher link to take you to the drupalci Jenkins instance.

Select the Build Artifacts link

Select the second half of the line - the 'artifacts' link
​​​​

Clicking on view will show you a complete build. Right click on the filename to save it locally (or copy the url and wget/curl it from somewhere.
The __orig file is there for when you submit a patch that makes changes to the drupalci.yml file, that should show you what it was before your new changes were applied.