Create version of run-tests.sh tuned for running PHPUnit

We may consider the use of ParaTest.

I tried to experiment a bit with paratest earlier and again now.

Unit tests

$ time ./vendor/bin/paratest -c core/phpunit.xml core/tests/Drupal/Tests/Component/
Time: 14.5 seconds, Memory: 16.00MB
Tests: 5540, Assertions: 11087, Incomplete: 7.
       14.59 real        38.34 user        16.12 sys

$ time ./vendor/bin/phpunit -c core/phpunit.xml core/tests/Drupal/Tests/Component/
Time: 24.91 seconds, Memory: 82.00MB
Tests: 8932, Assertions: 11087, Skipped: 1917.
       24.98 real        16.14 user         1.65 sys

I haven't done the research yet why it counts the amount of tests different nor why the pure phpunit runner skips some tests for me.

Kernel tests

$ time ./vendor/bin/paratest -c core/phpunit.xml core/tests/Drupal/KernelTests/Core/Entity/
Time: 2.01 minutes, Memory: 10.00MB
OK (162 tests, 5174 assertions)
      120.68 real       304.35 user        32.02 sys

$ time ./vendor/bin/phpunit -c core/phpunit.xml core/tests/Drupal/KernelTests/Core/Entity/
Time: 7.01 minutes, Memory: 12.00MB

OK (162 tests, 5174 assertions)
      421.12 real       267.67 user        20.72 sys

Browser tests

$ time ./vendor/bin/paratest -c core/phpunit.xml core/tests/Drupal/FunctionalTests/Routin
g/
Time: 33.33 seconds, Memory: 6.00MB
OK (4 tests, 13 assertions)
       33.35 real        41.35 user         2.56 sys

$ time ./vendor/bin/phpunit -c core/phpunit.xml core/tests/Drupal/FunctionalTests/Routing/
Time: 53.53 seconds, Memory: 6.00MB
OK (4 tests, 13 assertions)
       53.56 real        40.79 user         2.29 sys

TL;DR It speeds things up indeed

As a side note: https://github.com/symfony/symfony/blob/master/src/Symfony/Bridge/PhpUni... provides some poormans parallelism.

To be honest though run-tests.sh does not just solves the paralleism side of things, it opens up the result in a browser. It does things like

// Ensure all test locks are released once finished. If tests are run with a
// concurrency of 1 the each test will clean up its own lock. Test locks are
// not released if using a higher concurrency to ensure each test method has
// unique fixtures.
TestDatabase::releaseAllTestLocks();

The version of run-tests.sh that runs phpunit tests concurrently is run-tests.sh. :-)

$ php ./core/scripts/run-tests.sh --concurrency 32 --types="PHPUnit-Unit,PHPUnit-Kernel,PHPUnit-Functional"
## We run PHPUnit-FunctionalJavascript separately because it can't do concurrency.

The issue is maintainability and testability, not capability. See for instance #2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate where we start to make this hunk of code more maintainable. #2624926: Refactor run-tests.sh for Console component. tries to refactor the whole thing, which might not be the best idea in just one go. #2866082: [Plan] Roadmap for Simpletest talking about decoupling run-tests.sh and other testing infrastructure from the simpletest module.

Even if we do switch to paratest or whatever, we'll still need to maintain run-tests.sh because the testbot needs it for BC on contrib. So let's just make run-tests.sh maintainable.

Adding tag and parent because you can't actually search for 'run-tests.sh' on d.o.

As a side note, the performance of tests can be improved by adjusting the number of test processes to run in parallel.

Here are ParaTest results against core/tests/Drupal/KernelTests/Core/Entity made on my local environment (8 cores).

Given the above note (#4) there is no point in adding ParaTest to core unless it is considered as successor of run-tests.php in Drupal 9.

I have a hard time seeing us maintaining simpletest and run-tests.sh and all the suite in case phpunit totally cuts it. In general it feels like its better to make the phpunit stuff better than to iterating on the simpletest stuff, but you know, that's simply my personal opinion.

OK, so what specifically is missing that we'll get by going from three test runners to four?

I think it should be good if we'd be able to fix it before D9.

What needs fixing?

I meant get rid of run-test.sh and build/use a new phpunit based script which allows concurrency for drupalci.

run-tests.sh currently provides concurrency for PHPUnit-based tests under DrupalCI.

I don't love run-tests.sh, but we have two options:

Make run-tests.sh maintainable or re-write run-tests.sh.

The former can be accomplished in a way that's consistent. In fact most of the work has already been done and awaits review, or at least a ping of interest from anyone to do a reroll. #2866082: [Plan] Roadmap for Simpletest