[policy] Nightly regression testing for full contributed projects against both the production and development minor branches

Problem/Motivation

Currently, contrib tests run against the latest core version. Contrib projects may specify that a particular environment may run as "Issues & on Commit", "Branch on Commit", or "Branch Daily", but these options are always for the latest minor.

Contrib may trigger an individual test run against a different core branch using the "custom parameters", but this test run can only be triggered manually and does not become available for regular regression testing:

(Image courtesy @tim.plunkett.)

Core's minor version cycle and policies are intended to allow API additions, internal BC breaks, disruptions, UI changes, etc. in the scheduled minor versions, with only low-risk bug fixes in patch versions. However, the current testing policy means that contrib cannot take advantage of these scheduled disruptions, and is forced to track the development minor or run with broken HEAD. Something as small as changing a button text from "Save configuration" to "Save" can cause a contrib test suite that uses that form to fail, even though such a change is totally appropriate for a minor version of core. This is a frustration for contrib developers, and also makes it difficult to manage core policy for disruptions or additions to the testing API, because no matter what we do, contrib is broken immediately.

https://www.drupal.org/drupalorg/docs/drupal-ci/drupalorg-testing-policy does include a framework for triggering contrib tests against such targets, and identifies the following contrib targets (among others):

N - Semantic contrib patch-dev (2.0.x-dev)
N - Semantic contrib minor-dev(2.1.x-dev, 2.2.x-dev when 2.1.x-dev is in beta)
N - Semantic contrib major-dev(3.0.x-dev)

However, the policy does not currently cover the policy for which core code targets a contrib test can run against, and the current infrastructure does not yet support it.

There are concerns about scalability with contrib being able to specify too many frequent targets, so we discussed allowing only one environment for issues & on commit. However, having regular testing against the core versions the module supports -- both for the patch release and the minor -- is arguably more important than allowing contrib to test regularly against other databases/PHP versions/etc.

Finally, a contrib project branch might only support a certain minor version of core. my_contrib_project-8.x-1.x might work with 8.0.x, 8.1.x, and 8.2.x, while my_contrib_project-8.x-2.x might be compatible with 8.3.x+. Both 8.x-1.x and 8.x-2.x need to be tested for their respective branches until 8.3.0 is released.

Proposed resolution

Allow contrib projects to specify the desired testing scenario for the core development minor and the core stable branch, for the primary development environment only (the PHP version and database that core tests on patches and commit by default, currently PHP 5.5 / MySQL 5.5 but might be moved to PHP 7 / MySQL 5.5 in the future). This would appear at the top of the table on the QA page. The contrib maintainer would be able to select "Branch daily", "Branch on commit", or "Issues & Branch on commit" for this one environment for both relevant core branches (as rows) in the column for each of the contrib project's branches.

Remaining tasks

• Contrib maintainers can now specify tests against a specific branch of core in the new, more flexible testing interface thanks to #2651208: More-flexible, while reasonable, configured tests.

This unblocks the underlying concern of this issue.

However, this issue remains open because:

• It would be helpful to create default labels that chase core for 'current stable' and 'current dev' so that contrib maintainers don't have to manually configure the tests as core moves to keep up with testing.
• It may also be cost effective to limit these extra tests to the primary development environment, however, we did already implement a limit to a single environment for issue/patch testing. This wider branch testing is only available for the nightly tests configuration.