Problem/Motivation

Opening this as a core issue because it's 1/3rd core release cycle, 1/3rd contrib developer experience, 1/3rd DrupalCI.

https://www.drupal.org/core/d8-bc-policy and https://www.drupal.org/core/experimental mean that while 8.x minor releases should not have API changes as such, they're allowed to include changes that will break contrib modules relying on @internal APIs or functionality provided by experimental modules.

This allows us to actually improve 8.x in minor releases - depending on how strictly you interpret 'API change' we'd not be able to make 90% of changes we are otherwise, however it also means that sometimes core commits will cause contrib tests to start failing or actually break the module.

From core's perspective, we want to know if a specific change has broken one or more contrib modules as quickly as possible, since in some cases where a 'theoretical API change' turns out to be something lots of people were relying on, we may want to roll a patch back or add some kind of bc layer. This doesn't need to be automatic, active module maintainers are usually pretty quick to realise and open a core issue.

From contrib's perspective there are conflicting needs:

  • Contrib patches should probably always be tested against the current stable core branch - just because the minor version has a breaking change, doesn't mean the module is broken against the stable branch. It's my understanding that at the moment patches are tested against 8.1.x, longer term I think we should only make that switch when we get to release candidate stage.
  • Equally, it would be useful for contrib branch tests to be run against all active branches of core, since that notifies maintainers (once) that a core change has introduced a test failure
  • We do not have, and also do not want, support for multiple core versions in contrib modules. Since 8.0.x goes EOL as soon as 8.1.x is released, ideally contrib maintainers should not have to think about multiple core branches too much unless they're actively tracking something in the development branch, or specifically need to make a change during release candidate (at which point no more 8.0.x patch-release compatibility is needed since it's EOL by the time the new module release goes out).

    • Proposed resolution

    Not sure, just wanted a central place to discuss this since it's come up in irc several times.

    For me I think:

    - patches should only be tested against one core branch, we should increment the core branch when it gets to release candidate or potentially beta. For now that means no change until 8.2.0-beta1 or 8.2.0-rc1 since we're already testing against 8.1.x and no point changing it back

    - branches should be tested against both the current minor and the development minor. I don't think we need to worry about 8.2.x until after 8.1.x is out, so maximum of two branches at a time at least for now. Also this only needs to be every 24 hours at most.

    Remaining tasks

    User interface changes

    API changes

    Data model changes

Comments

catch created an issue. See original summary.

catch’s picture

catch
he/him
Primary language English
CreditAttribution: catch at Third and Grove commented
Issue summary: View changes

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

drumm’s picture

drumm
he/him
Location NY, US
CreditAttribution: drumm at Drupal Association commented

PIFT currently uses the most recent dev release shown on https://www.drupal.org/project/drupal. And you can do one-off custom tests with other versions.

The direction we're thinking for #2651208: More-flexible, while reasonable, configured tests (updating that issue is on my todo list) is having an explicit default configured, and letting project maintainers choose from

  • Default, currently drupal 8.1.x
  • 8.0.x
  • 8.1.x
  • 8.2.x

when configuring tests for their project. We can move the default as appropriate. This issue can focus on a policy for what is the default at any given time.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
CreditAttribution: mile23 as a volunteer commented

Just want to point out how @deprecated test classes bork contrib during a minor version change: #2690403: Fatal error: Class 'Drupal\field\Tests\FieldUnitTestBase' not found

Basically my contrib uses Drupal\field\Tests\FieldUnitTestBase which was radically changed in #2684095: Convert field kernel tests to KernelTestBaseTNG leading to a period of time during which all patches fail until the maintainer restarts them with the proper core version.

(from issue summary):

Contrib patches should probably always be tested against the current stable core branch - just because the minor version has a breaking change, doesn't mean the module is broken against the stable branch. [..]

Equally, it would be useful for contrib branch tests to be run against all active branches of core, since that notifies maintainers (once) that a core change has introduced a test failure

These are both generally correct, except that *right now* I have to educate my contrib participants that their failed test is actually a passing one because they have to select the proper core branch.

Having a way to say 'Always test patches against core stable and once a week test against every dev' would be super-awesome.

We do not have, and also do not want, support for multiple core versions in contrib modules.

This contrib maintainer wants to pin his contrib to a stable release. In an ideal world this would mean specifying semver in the info.yml file. If my contrib were pinned to, say, 8.0.* and 8.1.0 was released and 8.0.x were marked unsupported, then drupalci should fail my branch test.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

quietone CreditAttribution: quietone at PreviousNext commented

There has been no discussion here for 8 years but there has been progress in this area.

Drupal is now using GitLab for testing and there are issues in the GitLab templates project on this topic. There is also a wiki page for how to use GitLab CI. One section of which explains how to test the previous and next major release and the previous and next minor release, which is the proposed resolution of the issue summary. Therefore, I think this can be closed.

quietone’s picture

quietone CreditAttribution: quietone at PreviousNext commented
Status: Active » Closed (outdated)

catch agrees with me that this can be closed. He also pointed out that it is 'nice that it's all possible now'. And yes, it is.

quietone’s picture

quietone CreditAttribution: quietone as a volunteer and at PreviousNext commented
Version: 11.x-dev » 10.3.x-dev

Changing to latest version when this was closed.