Problem/Motivation
Core gitlab runs use run-tests for concurrency, on only one test suite at a time (unit, kernel, functional, functional js).
In contrib gitlab runs, concurrent test runs run all groups at once, and this appears to go in the unit, kernel, functional, functional js order.
See the order of initial output on:
https://git.drupalcode.org/project/paragraphs/-/jobs/629259
i.e. If we run functional javascript -> functional -> kernel -> unit instead, we should be able to get overall faster test runs.
This is because functional and functional js tests take dozens or hundreds of times as long to run as unit tests, so if concurrency is maxed out, several unit tests can be finished in the space of one functional test. If a functional test is already running and there are spare threads for unit tests, this time is 'free', but if the unit tests all run first, then you end up with long running tests occupying just a single thread at the end of the job, with other CPUs idle.
Steps to reproduce
Proposed resolution
When multiple tests are run by run-tests.sh, reverse the order of the test groups so that slowest tests run first.
We will need to decide whether @group #slow should be used to determine the order only within a test group, or the order of tests within the whole run - it might make sense for it to be within the whole run?
If we only change the default order of test discovery, @group #slow will continue to work to modify the order of tests regardless of type, which seems fine and makes this a very small change.
Remaining tasks
This will have no effect on core test runs, so ideally we'd see how much difference it makes with a contrib module that has a decent test suite like paragraphs.
User interface changes
API changes
Data model changes
Release notes snippet
Issue fork drupal-3449743
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3449743-optimize-test-order
changes, plain diff MR !8181
Comments
Comment #2
catchComment #4
catchComment #5
smustgrave commentedSo seems like one of those features that may have to be merged first to truly measure impact right?
Comment #6
berdirI think it would be possible to create a merge request in a module like webform or paragraphs and apply the core patch and verify the output/execution time before/after. But we can also do that after this was merged.
Comment #7
larowlanCommitted to 11.x and backported to 11.0.x, 10.4.x and 10.3.x
Also backported to 10.2.x even though technically this is a task because for contrib modules testing this, they will still be using the 10.2.x checkout for some time. If folks disagree with that, happy to wind that back.