Problem/Motivation

Projects on D.O currently use an arbitrary the most recent dev branch shown on the Drupal 8 project page branch as default checkout for contrib CI testing.

This means that I can't automatically test my Drupal 8.0.x contrib against 8.0.x because D.O defaults to 8.1.x at some point in the CI chain.

Proposed resolution

As a default, project-issue should use the current issue version setting to determine which branch to check out and test against.

This information can then be fed to drupalci_api for CI.

Remaining tasks

User interface changes

API changes

Data model changes

Comments

Mile23 created an issue. See original summary.

drumm’s picture

Project: Project issue tracking » Project issue file test
Component: Tests » Code
Issue summary: View changes
Related issues: +#2651208: More-flexible, while reasonable, configured tests
wim leers’s picture

mglaman’s picture

Right now this is causing Profile to fail, and any other modules which use the Entity module. Here is a job that shows the problem: https://www.drupal.org/pift-ci-job/213633. Modules like Entity and Chaos Tools are incubators for core, especially since the new release cycle allows for more rapid changes.

An enhancement in Entity module was committed to 8.1.x (or maybe 8.2.x.) However, the code was not removed from the module, yet, because 8.1.x does not have a release. This is causing DrupalCI to fail for Profile, resulting me to go back to Travis.

In my opinion, it would be best to allow a contributed project to specify what stable minor version it supports. This could even be useful on the project's page "developed and test up until 8.x.x" so users know what minor version it was last worked with.

This could help users in deciding on whether to upgrade to a minor version, or if they have dependencies not yet ready (although minor -> minor shouldn't be that difficult.) Unless it's a module like Entity.

mile23’s picture

Note that currently you can at least re-run the test with an arbitrary branch.

mile23’s picture

catch’s picture

I opened #2680037: [policy, no patch] Figure out core branch testing needs for contrib modules against core, not knowing about this issue, but they're similar.

This could help users in deciding on whether to upgrade to a minor version, or if they have dependencies not yet ready (although minor -> minor shouldn't be that difficult.) Unless it's a module like Entity.

Once 8.1.0 is released, 8.0.x support is dropped. We're trying to leave a 2-4 week window before the next security release window to give people time to update, but theoretically there is no case where a contrib module would be expected to work with two stable releases at once. The one exception might be a security update to a contrib module on the day 8.1.0 is released or similar.

So the main issue we have is when contrib modules move to testing against the new development branch by default. For me, especially now we have elective testing against other branches, I'd go for when the release candidate comes out.

So taking 8.2.x, when we release 8.2.0-rc1 we'd switch all contrib testing to default to that branch - that's the last chance to report unintended API changes etc. Module maintainers wanting to get ahead, could either run tests manually or there could be a default branch option in the UI eventually. But at some point we need to migrate everyone to the new minor even if there's a UI.

drumm’s picture

Status: Active » Postponed

Postponing on #2651208: More-flexible, while reasonable, configured tests. We need to refactor the UI in general to even have screen real-estate for this.

mile23’s picture

Once 8.1.0 is released, 8.0.x support is dropped. We're trying to leave a 2-4 week window before the next security release window to give people time to update, but theoretically there is no case where a contrib module would be expected to work with two stable releases at once.

This might be the case if it weren't test classes that were deprecated: #2690403: Fatal error: Class 'Drupal\field\Tests\FieldUnitTestBase' not found

So basically Examples is written against 8.0.x, which is the current stable release. I have to be able to specify that in the CI process, or else all tests will die until I restart the tests using 8.0.x branch, even before 8.1.x or 8.2.x is released.

Alternately, I can swap my tests to use the newer tests so they'll be able to run against the default 8.2.x CI, but then I'm working against 8.2.x instead of the current stable, which is what I want.

This is currently what's happening in my issues.

If I could specify which of my branches correspond to core branch, then I'd love to be able to do that. Unfortunately, we don't have semver for the core: value of the info.yml file, or AFAIK any naming convention for contrib branches.

Mixologic’s picture

Component: Code » Testing Configuration
drumm’s picture

Assigned: Unassigned » drumm
Status: Postponed » Closed (duplicate)

#2651208: More-flexible, while reasonable, configured tests will fix this. Project tests can be configured to be pinned to a core branch, or follow the default which is the most recent dev branch shown on the project page.