Problem/Motivation

run-tests.sh is pretty much the single most important piece of code in the Drupal QA process.

It's basically impossible to search for the string 'run-tests.sh' so it's difficult to be a dev who wants to work on it. :-)

This issue is one to use as a parent if you make an issue about run-tests.sh, in order to overcome the lack of searchability.

I've also taken the liberty of adding an issue tag for run-tests.sh, which might also help people find their way around.

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Comments

Mile23 created an issue. See original summary.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

neclimdul’s picture

neclimdul
He/Him
commented

Is this the place to voice the opinion that as we move to phpunit, run-tests.sh becomes a drupalci artifact that probably shouldn't be in core?

dawehner’s picture

dawehner
Primary language German
commented

Is this the place to voice the opinion that as we move to phpunit, run-tests.sh becomes a drupalci artifact that probably shouldn't be in core?

That's a valid question. Let's assume we don't have any simpletests anymore, would we need run-tests.sh to be able to run anything beside an entire testsuite (Kernel, Functional etc.)?
Basically run-tests.sh could end up with a wrapper which catches fatals inside phpunit itself and expose that to drupalci.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

Is this the place to voice the opinion that as we move to phpunit, run-tests.sh becomes a drupalci artifact that probably shouldn't be in core?

See #2750461-28: Remove Simpletest UI because we don't want to maintain a graphical test runner

neclimdul’s picture

neclimdul
He/Him
commented

re #4yeah pretty much. something something parallel tests as well maybe.
re #5your link in that that issue is why I'm here. I'm all for fixing up run-test.sh, the question clearly wasn't relevant to the Web UI discussion though. It is relevant to deciding how we look at the process of cleaning up run-tests.sh though. e.g. what is our audience.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

We tell people to run run-tests.sh when they say 'it passed locally..'

We also use run-tests.sh to isolate phpunit fatals.

drupalci is tooled to use run-tests.sh, but we can modify that as needed, because drupalci is more flexible than it looks.

If we take away the simpletest UI, then the onus is on run-tests.sh to show results of test runs.

I was trying to figure out the best set of small steps to take. I think refactoring existing functionality to make it more maintainable also makes it more flexible for us to make more major decisions. So for instance if we were to take all the PHPUnit runner stuff from simpletest module and move it to a class/library/component, that would aid us in either keeping run-tests.sh and maintaining it, or eventually writing or adapting a better runner and removing run-tests.sh. (That's this one: #2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate which isn't really a run-test.sh improvement, though it does touch it.)

There are plenty of other examples of small improvements we can make that knock loose some cobwebs to make it easier to change in any direction later.

Because as a community we don't seem to be able to decide what to do, so we're left with an unmaintainable pile of code, which prevents us from moving forward on anything, which keeps us from making decisions. :-)

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

New child: #2905007: Allow run-tests.sh to report skipped/incomplete PHPUnit tests

neclimdul’s picture

neclimdul
He/Him
commented

"We also use run-tests.sh to isolate phpunit fatals." Maybe its because I'm using php 7 exclusively now but I couldn't find a fatal error that wasn't handled perfectly by phpunit.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

If you have three test classes and one of them passes and one crashes, then phpunit won't be able to tell you which one passed and which one didn't run. You'll have an error log but not a report.

neclimdul’s picture

neclimdul
He/Him
commented

I'm saying that is not the case. The junit output correctly documents the fatal error.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

'Crashes' as in PHPUnit fatals, not that the test reports an error. run-tests.sh works around it by making a template fail report: #2560643: Fatal PHPUnit tests are not always reported as failure

neclimdul’s picture

neclimdul
He/Him
commented

I think currently that just double reports failures.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

Want a coverage report from run-tests.sh? #2974911: Allow run-tests.sh to generate coverage reports

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

andypost’s picture

andypost
he/him
Primary language Russian
commented
Version: 8.6.x-dev » 8.8.x-dev

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.