Once #3465602: Order tests by number of public methods to optimize gitlab job times lands we have tests distributed fairly evenly between test jobs on the gitlab pipeline.
I've also been experimenting in a sandbox branch with #3416522: Add the ability to install multiple modules and only do a single container rebuild to ModuleInstaller and #3469309: Use one-time login link instead of user login form in BrowserTestBase tests + various individual test performance issues combined to see what the absolute potential floor of test run time is.
With all of those applied, the best run I've managed is 4m55s which is down from the current floor of about 5m30s.
https://git.drupalcode.org/project/drupal/-/pipelines/261246
However, that time was achieved with much lower overall CPU requirements than the current pipelines - this issue is to extra those changes from the sandbox MR, however it will depend on some of the other issues landing in order not to be a regression against the current state (at least in terms of wall time).
Show commandsStart within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes, plain diff MR !9302changes, plain diff MR !9561
Comments
Comment #2
catchComment #4
catchWhat this does:
1. Lowers the CPU request from 24 to 16 for most jobs. The theory behind this is that the total-CPUs-per-machine is more likely to be a multiple of 16 than 24 so theoretically we can fit more jobs on a lower number of machines (or on 16 CPU machines if such a machine exists). I don't fully (or even much) understand the relationship between CPU requests, kubernetes and AWS instances, so this might be flawed, but also in general lower and simpler numbers seems better.
e.g.
16 + 16 + 16 + 16 = 64
24 + 24 + 24 = 72.
2. Lowers the concurrency of a couple of jobs quite a lot, especially functional tests where I am pretty sure the concurrency in HEAD is leading to CPU contention and hence slower rather than faster test runs. This is made possible by #3465602: Order tests by number of public methods to optimize gitlab job times which removes @group #slow from the vast majority of tests, relying on a better ordering algorithm instead.
3. Increases the parallelism for functional js and functional tests by 1 each. This is because in theory most test runs (in the sandbox branch with various changes applied) can finish within about 2m30s, but we still have a lot of individual tests over 2 minutes each. With lower concurrency, those long running jobs are spread out enough we don't run two slow tests end to end. I'm pretty sure there is potential to bring this lower by continuing to optimise some of these slower tests, but it also gives us a bit of headroom when we add new coverage.
If we look at the jobs, we can see that the overall CPU requirement is reduced dramatically:
Functional JS:
Before: 2 * 24 = 48
After: 3 * 16 = 48
Functional:
Before: 7 *24 = 168
After: 8*16 = 128
W3 legacy:
Before: 1 * 24 = 24
After: 1 * 16 = 16
So an overall reduction of 48 CPUs per pipeline, with potential scope to reduce further.
Comment #5
catchFound an extra 16 CPU requests to drop on #3465602: Order tests by number of public methods to optimize gitlab job times which brings the total to 64 combined with the changes here.
Comment #6
catchJust rebased and the full pipeline took 6m5s: https://git.drupalcode.org/project/drupal/-/pipelines/287574
Our current best runtime is currently around 5m30s, given the amount of variation between runs, that seems in range given the overall CPU saving here. Moving to needs review.
Comment #7
fjgarlin commentedThe changes look good and so do your maths in #4. Pipelines are also happy. RTBC.
Comment #9
longwaveCommitted 8877452 and pushed to 11.x. Thanks!
Patch doesn't apply to 11.0.x and below, we don't run as many tests there, but this feels like a good candidate for backport if it reduces costs for the DA?
Comment #10
catchI'm not sure it's worth backporting to 11.0.x but it probably is worth backporting to 10.4.x since that will then carry forward to the next 10.x branches which will have daily test runs for another couple of years.
10.4.x does not have all of the test performance improvements in 11.x, but I'm sure that tests would still finish in 7-8 minutes or less with these changes, and we don't run that many MR pipelines against 10.4.x (compared to on-commit/scheduled runs).
Also if my uninformed kubernetes theories are correct, it might help recycling/re-use of test runners since they'll be more consistent requests between the branches?
So moving back there. If there's a problem with 10.4.x and the changes here, we'll find out from the backport pipeline hopefully.
Comment #13
catchBackport pipeline finished in 6 minutes and 12 seconds. https://git.drupalcode.org/project/drupal/-/pipelines/288832
Since the backport itself was trivial, going to go ahead and commit here.