Problem/Motivation

Now that migrate_drupal and some other modules are being removed from core, we have less total tests to run, so can try to reduce CPU requests for jobs.

Doing this requires a bit more fine tuning of which tests are marked slow vs. not in order to maintain quick wall times for pipelines.

Steps to reproduce

Proposed resolution

Reduce the CPU request by 2 each for kernel and functional js jobs, and by 4 for functional jobs.

Increase the CPU request and concurrency for build tests (still one job, just more tests at once).

Because we still have a lot of individually slow tests, not trying to reduce test concurrency too much - we still need the slow tests to all start asap.

This also makes these jobs depend on phpstan, which was dropped when we flattened the pipeline structure. Reduces the threshold for PHPStan to create parallel jobs so that MRs touching 21-79 files get 2-3 processes instead of one. On this MR specifically it shaves about 20s off the job time without increasing the CPU request.

Moves the build job to be the first job that runs because it's the slowest (moving jobs in the YAML affects running order, but not order in the UI which is alphabetical).

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3578719

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

catch created an issue. See original summary.

catch’s picture

Title: Kernel test running tweaks » Test running tweaks
Status: Active » Needs review

This turned into a bit of an omnibus issue but seems to be getting somewhere.

It looks like we can reduce the CPU request and concurrency for functional tests without a noticeable hit (possibly no hit at all) on overall runtimes. The trade-off is this makes balancing between runners a bit more finely balanced, so I moved #slow around various tests.

This morning's pipelines seem to have landed on fast gitlab runners and this afternoon's on slower runners (or maybe more sharing when it gets busier, never sure), so it's incredibly hard to be confident overall.

catch’s picture

Status: Needs review » Active

I looked into reducing cpu/concurrency for kernel/functional js tests but doesn't look like we're quite there yet. Maybe we could do something like reduce from 16 to 14 CPUs but that feels a bit weird to write.

catch’s picture

Status: Active » Needs review
catch’s picture

Title: Test running tweaks » Reduce minimum CPU requests for kernel, functional, and functional js requests
Issue summary: View changes
catch’s picture

Issue summary: View changes
catch’s picture

I think I found a phpstan bug or at least quirk.

It has a configuration option:

minimumNumberOfJobsPerProcess

This defaults to 2. There is also a jobSize of 20 files per job. This MR touches 58 files that PHPStan would scan.

If it spawned two process, one would have 2 jobs of 40 files, and one process would have 1 job of 18 files (or 38 and 20 but same difference). So I think because the minimum is 2, it spawns only one job and gives it all 58 files.

This means any number of files between 21 and 60 will only get one process, even when there are 3 other CPUs just sitting there. 21-60 files covers quite a lot of core MRs.

See https://git.drupalcode.org/project/drupal/-/jobs/9014066

Parallel processing scheduler:
# of detected CPU cores:   72
# of analysed files:       58
# of jobs:                 3
# of spawned processes:    1
Elapsed time: 48.17 seconds

vs: https://git.drupalcode.org/project/drupal/-/jobs/9014835

Parallel processing scheduler:
# of detected CPU cores:   64
# of analysed files:       58
# of jobs:                 3
# of spawned processes:    3
Elapsed time: 19.50 seconds

In the process of researching that, I also found https://github.com/phpstan/phpstan/issues/7479 which suggests the next phpstan release will fix CPU detection on kubernetes.

Trying jobSize: 10 now to see if that drops an extra 10 seconds off..

edit: jobSize 10 doesn't seem to help - possibly the overhead of each process is enough to cancel out doing less files.

catch’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
catch’s picture

Issue tags: +Test suite performance

With that phpstan change I got pipeline times of 5.5m and 6m the past two runs, lowest ever is 4m20s recently but that's very, very rare to get under 5 minutes and depends on runners sitting there waiting etc. Usual runs are 5.5-8 minutes so we're still at the lower end of that.

catch’s picture

Kicked off another pipeline to make sure things are consistent enough.

PHPStan looks like this:

Parallel processing scheduler:
# of detected CPU cores:   48
# of analysed files:       61
# of jobs:                 4
# of spawned processes:    4
Elapsed time: 21.99 seconds

Whole job finished in 6m8s. Slowest individual job was the build test which is getting extra resources here rather than less. https://git.drupalcode.org/project/drupal/-/jobs/9018605 (but also runs much faster than that sometimes) every other job was at least 15s faster than that.

So for me this is about the right balance for everything, we can always fiddle more later on.

mondrake’s picture

Status: Needs review » Reviewed & tested by the community

Looks good to me.

  • godotislate committed 20c1667f on main
    test: #3578719 Reduce minimum CPU requests for kernel, functional, and...
godotislate’s picture

Status: Reviewed & tested by the community » Fixed

Committed 20c1667 and pushed to main. Thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

benjifisher’s picture

Title: Reduce minimum CPU requests for kernel, functional, and functional js requests » Reduce minimum CPU requests for kernel, functional, and functional js tests

I think we mean "tests", not "requests", at the end of the title. I hope I am right.

catch’s picture

Yes.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.