Run contrib module branch tests against both dev and latest stable core branches

This can be done in two phases:

First - Make sure testing selects the one correct current version to test against. This removes the blocker, and is all we need until 8.1.x is branched.

Second - Testing against multiple core branches. I think this may only be supported with the new drupalci stack, along with other multi-environment testing.

adding "infrastructure blocker for Drupal 8.0.0" tag, since this blocks the drupal 8 core issue: #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1, so that the infrastructure blockers to d8 release (in issue queues outside of the core queue) is accurate. Remove the blocker tag when this issue is fixed, and update 2267715.

Would we need to run the branch tests against every environment for *both* dev and latest stable, or would acceptable to use just the 'default' testing environment (probably mysql 5.5/php5.5) for contrib branch tests run with the dev branch of core?

Could we make the default test environment(s) selectable per contrib branch? By default test against the default (php5.5/mysql5.5) and add environments to default test against.
I think the default (php5.5/mysql5.5) is enough for most, but for some projects it could be useful to set another / extra default test environment (thinking for example db driver projects). Could be a followup perhaps to set additional default test environments.

@mallezie - that already exists - contrib can specify which environments that branch, commit, and patch tests are ran against. This issue is about what branch of *core* is checked out for those contrib tests.

There is another wrinkle in this, however, and that is if we get to a state where we are using composer to resolve core's dependencies, then we might need to also run the tests against both versions of core (dev and stable) by their minimum and maximum supported dependency versions.

This ultimately blocks 8.0.0, so escalating to critical.

Two parts to solving this issue:

• Testing against 8.0.x is hard-coded right now. We can always change the hard-coding at the right time, but it really shouldn't be hard-coded.
• Actually adding the option to test against a non-default core branch.

For part one, there currently isn't a programmatic way for Drupal.org to know whether 8.1.x or 8.0.x is current. #2385781: Replace "Snapshot" on Releases page for projects with per-release field will add a field that PIFT can use.

With #2385781: Replace "Snapshot" on Releases page for projects with per-release field now deployed, Drupal core can branch 8.2.x as a placeholder, make the release, unchecking "Show in project’s download table when branch is supported". That checkbox will do what it says, and issues can be moved to the 8.2.x version.

And PIFT can use this to know that the most recent shown in the download table is the current core branch.

Next step to solving this issue: #2627450: Do not hard-code 8.0.x for contrib testing

I believe that takes care of the prerequisites. Now we need a UI for selecting the core branch for tests.

Here is the UI so far for adding a one-off test for a contrib project's branch:

Should the UI also indicate which core branch each test is tested against? I.e., I assume the existing tests listed for CTools are 8.0.x and the added one would be in a similar pill.

Yep, I've added that, along with storage for the branch.

The 7.x-3.x commits are now deployed on Drupal.org. New contrib tests will now show the core branch used in the UI.

On the feature brach - the custom test UI is hooked up to the data storage. The missing part is actually listing the custom tests for the project so you can find them again. And the same for issue tests.

The UI from #22 is now deployed on Drupal.org. Next up is the same for issue testing.

I put in some improvements to the project testing listing too. Now shown:

• Most recent completed test for every branch/environment.
• Any queued tests.
• Custom tests created in the last 30 days.

One additional suggestion -- it was not clear to me that I had to check "Custom parameters" to get the selectors for the branch and environment. Is there any way to improve the label for that checkbox, or maybe just show those selectors all the time?

Good idea, here's what changed:

• Added white space above the custom parameters checkbox, it is distinct enough from the other options.
• Disabled the select options instead of hide when unchecked.
• Added a fieldset to group them under the checkbox.

I'm going to be bold and see if this is fixed. It can be deployed to Drupal.org tomorrow.

This does only cover one-off testing with non-default core branches, not regular automated testing. This issue did get a big chunk of work done, adding the concept of custom tests, which I expect we’ll expand in the future. Making this available as a default option, like “test … with 8.0.x daily,” will need a re-think of the table-of-selects used to configure project testing currently. That probably needs a separate issue.

This has been deployed to Drupal.org.

Nice. :-)

Heres' a bit of a follow-up to cover regular automated testing: #2663018: Useful core branch defaults for contrib CI