Problem/Motivation

run-tests.sh splits test classes between parallel CI jobs with WorkAllocator. It deals out #slow classes round robin, then the other classes the same way. Round robin ignores how much work each job already has. Each round gives the largest class to the first job, and both the slow and normal passes start at the first job, so the first jobs get more work.

Steps to reproduce

Proposed resolution

Each test class now gets a weight: its test case count, multiplied by 8 for #slow classes. Classes are then sorted by weight, heaviest first. Each class goes to the job with the lowest total weight so far. Ties are broken by class name so sorts are consistent.

With the help of Claude Code, a simulation took test run times from three daily main pipelines (996748, 997801 and 999018). Mean time of the slowest job, before and after:

  • Kernel: 146 s to 143 s
  • Functional: 236 s to 203 s
  • FunctionalJavascript: 252 s to 238 s

EntityDefinitionUpdateMultipleTypesTest takes 128 seconds which sets the floor for kernel tests.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3630648

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

longwave created an issue. See original summary.

longwave’s picture

Issue summary: View changes
longwave’s picture

Status: Active » Needs review

I think we can also safely reduce the number of kernel jobs from 4 to 3, as functional test jobs all take significantly longer.

longwave’s picture

https://git.drupalcode.org/project/drupal/-/pipelines/1000957 completed in 5m19

Kernel tests took 143s, 118s, 128s
Functional tests took 123-255s
FunctionalJavaScript tests took 215s, 253s, 215s

We could even try reducing kernel to 2, but I think in some cases that will make it run slightly longer.

mondrake’s picture

Adding some related issue in this 'test work allocation optimization' space that have similar intent, ftr and later ease of reference.

I think this is good, probably we just need to try it and see its performance. The test coverage is also good to ensure isolation between the bins.

The only thing I would suggest here is to add a 'weight' column to the debug-discovery report, near to the 'count' column, so to make more visually clear the logic for the sorting, as this is no longer obvious.

Test execution sequence. Tests marked *** will be executed in this PARALLEL BIN #2.
-------------------------------------------------------------------------------------
    Sort  Bin                                 
Bin  Seq  Seq Slow? Group            Cnt Class
-------------------------------------------------------------------------------------
       1      #slow media             52 Drupal\Tests\media\Kernel\MediaAccessControlHandlerTest     
***    2    1 #slow package_manager   30 Drupal\Tests\package_manager\Kernel\SandboxManagerBaseTest  
       3      #slow Database          25 Drupal\Tests\mysql\Kernel\mysql\SchemaTest                  
       4      #slow Entity            22 …elTests\Core\Entity\EntityDefinitionUpdateMultipleTypesTest
mondrake’s picture

Issue tags: +run-tests.sh
longwave’s picture

Status: Needs work » Needs review

Added a weight column and simplified the code a bit.

We can try tweaking the magic number 8 but I'm not sure exactly how much difference it will make.

mondrake’s picture

Status: Needs review » Reviewed & tested by the community

Tweaking the weight of the #slow tests would probably make sense if we end up with major WT differences between bins of the same test type.

Let’s see first; time will tell. RTBC