Problem/Motivation

We currently have quite a few tests skipped due to known random failures. Skipping lots of tests have the positive impact of having more consistent test results, but has the downside of reduced test coverage. At the moment, there isn't documented easy way to run all of the skipped tests for changes that have higher risk of introducing regressions.

Proposed resolution

Move skipped tests to a new test group that is not executed by default on a test run. Document how to modify drupalci.yml to include the risky tests on the test run, so that changes that have higher risk of introducing regressions can be run against those.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Comments

lauriii created an issue. See original summary.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented

This seems like a good idea - tests can have more than one group and we can automagically skip the tests in WebDriverTestBase::setUp is they have this group.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented

One thing is that not all the tests that are skipped are skipped at the beginning so that'd make this hard... maybe we need to override markTestSkipped on WebDriverTestBase and make that not skip if you have an environment variable set.

lauriii’s picture

lauriii
he/him
Primary language Finnish
Location Finland
commented

That's a good point! The reason I thought about the group was that it would then also allow making a separate patch that would only run the risky tests which would make the test run faster.

Version: 10.0.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.