Problem/Motivation

run-tests.sh splits test classes between parallel CI jobs with WorkAllocator. It deals out #slow classes round robin, then the other classes the same way. Round robin ignores how much work each job already has. Each round gives the largest class to the first job, and both the slow and normal passes start at the first job, so the first jobs get more work.

Steps to reproduce

Proposed resolution

Each test class now gets a weight: its test case count, multiplied by 8 for #slow classes. Classes are then sorted by weight, heaviest first. Each class goes to the job with the lowest total weight so far. Ties are broken by class name so sorts are consistent.

With the help of Claude Code, a simulation took test run times from three daily main pipelines (996748, 997801 and 999018). Mean time of the slowest job, before and after:

  • Kernel: 146 s to 143 s
  • Functional: 236 s to 203 s
  • FunctionalJavascript: 252 s to 238 s

EntityDefinitionUpdateMultipleTypesTest takes 128 seconds which sets the floor for kernel tests.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3630648

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

longwave created an issue. See original summary.

longwave’s picture

Issue summary: View changes
longwave’s picture

Status: Active » Needs review

I think we can also safely reduce the number of kernel jobs from 4 to 3, as functional test jobs all take significantly longer.

longwave’s picture

https://git.drupalcode.org/project/drupal/-/pipelines/1000957 completed in 5m19

Kernel tests took 143s, 118s, 128s
Functional tests took 123-255s
FunctionalJavaScript tests took 215s, 253s, 215s

We could even try reducing kernel to 2, but I think in some cases that will make it run slightly longer.

mondrake’s picture

Adding some related issue in this 'test work allocation optimization' space that have similar intent, ftr and later ease of reference.

I think this is good, probably we just need to try it and see its performance. The test coverage is also good to ensure isolation between the bins.

The only thing I would suggest here is to add a 'weight' column to the debug-discovery report, near to the 'count' column, so to make more visually clear the logic for the sorting, as this is no longer obvious.

Test execution sequence. Tests marked *** will be executed in this PARALLEL BIN #2.
-------------------------------------------------------------------------------------
    Sort  Bin                                 
Bin  Seq  Seq Slow? Group            Cnt Class
-------------------------------------------------------------------------------------
       1      #slow media             52 Drupal\Tests\media\Kernel\MediaAccessControlHandlerTest     
***    2    1 #slow package_manager   30 Drupal\Tests\package_manager\Kernel\SandboxManagerBaseTest  
       3      #slow Database          25 Drupal\Tests\mysql\Kernel\mysql\SchemaTest                  
       4      #slow Entity            22 …elTests\Core\Entity\EntityDefinitionUpdateMultipleTypesTest
mondrake’s picture

Issue tags: +run-tests.sh
longwave’s picture

Status: Needs work » Needs review

Added a weight column and simplified the code a bit.

We can try tweaking the magic number 8 but I'm not sure exactly how much difference it will make.

mondrake’s picture

Status: Needs review » Reviewed & tested by the community

Tweaking the weight of the #slow tests would probably make sense if we end up with major WT differences between bins of the same test type.

Let’s see first; time will tell. RTBC

  • catch committed 71fee78d on 12.0.x
    task: #3630648 Improve balancing of CI test jobs
    
    By: longwave
    By:...

  • catch committed 6842d98b on main
    task: #3630648 Improve balancing of CI test jobs
    
    By: longwave
    By:...
catch’s picture

Version: main » 12.0.x-dev
Status: Reviewed & tested by the community » Fixed
Related issues: +#3615585: Optimize EntityDefinitionUpdateMultipleTypesTest

I have considered opening an issue to add ##slow and ###slow multiple times but assumed that people would point and laugh at me for it, this wouldn't necessarily stop me from doing it.

This looks great to me, good way to get both walltimes and CPU usage down at the same time without needing to refactor any tests at all!

We could also potentially look at boosting the weight for lower method-count slow tests a bit - those tend to have the slowest individual methods. e.g. assuming average method times of 7s, a test with one 20s method doesn't need #slow (although it's annoying if it's the last test to run, but usually not a big deal), a test with ten 20s methods generally does, a test with one 120s method definitely does. Might open a follow-up to experiment a bit, but easier to do that with this already in.

#3615585: Optimize EntityDefinitionUpdateMultipleTypesTest improves EntityDefinitionUpdateMultipleTypesTest performance and needs another review.

Committed/pushed to main and 12.0.x, thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

catch’s picture

mondrake’s picture