Problem/Motivation

Core gitlab runs use run-tests for concurrency, on only one test suite at a time (unit, kernel, functional, functional js).

In contrib gitlab runs, concurrent test runs run all groups at once, and this appears to go in the unit, kernel, functional, functional js order.

See the order of initial output on:
https://git.drupalcode.org/project/paragraphs/-/jobs/629259

i.e. If we run functional javascript -> functional -> kernel -> unit instead, we should be able to get overall faster test runs.

This is because functional and functional js tests take dozens or hundreds of times as long to run as unit tests, so if concurrency is maxed out, several unit tests can be finished in the space of one functional test. If a functional test is already running and there are spare threads for unit tests, this time is 'free', but if the unit tests all run first, then you end up with long running tests occupying just a single thread at the end of the job, with other CPUs idle.

Steps to reproduce

Proposed resolution

When multiple tests are run by run-tests.sh, reverse the order of the test groups so that slowest tests run first.

We will need to decide whether @group #slow should be used to determine the order only within a test group, or the order of tests within the whole run - it might make sense for it to be within the whole run?
If we only change the default order of test discovery, @group #slow will continue to work to modify the order of tests regardless of type, which seems fine and makes this a very small change.

Remaining tasks

This will have no effect on core test runs, so ideally we'd see how much difference it makes with a contrib module that has a decent test suite like paragraphs.

User interface changes

API changes

Data model changes

Release notes snippet

Issue fork drupal-3449743

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

catch created an issue. See original summary.

catch’s picture

Status: Active » Needs review

catch’s picture

Issue summary: View changes
smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

This will have no effect on core test runs, so ideally we'd see how much difference it makes with a contrib module that has a decent test suite like paragraphs.

So seems like one of those features that may have to be merged first to truly measure impact right?

berdir’s picture

I think it would be possible to create a merge request in a module like webform or paragraphs and apply the core patch and verify the output/execution time before/after. But we can also do that after this was merged.

larowlan’s picture

Version: 11.x-dev » 10.2.x-dev
Status: Reviewed & tested by the community » Fixed

Committed to 11.x and backported to 11.0.x, 10.4.x and 10.3.x

Also backported to 10.2.x even though technically this is a task because for contrib modules testing this, they will still be using the 10.2.x checkout for some time. If folks disagree with that, happy to wind that back.

  • larowlan committed 3956262a on 10.2.x
    Issue #3449743 by catch: Try to optimize test ordering when run-tests.sh...

  • larowlan committed 40b02150 on 10.3.x
    Issue #3449743 by catch: Try to optimize test ordering when run-tests.sh...

  • larowlan committed 463ffc70 on 10.4.x
    Issue #3449743 by catch: Try to optimize test ordering when run-tests.sh...

  • larowlan committed 0749ca2e on 11.0.x
    Issue #3449743 by catch: Try to optimize test ordering when run-tests.sh...

  • larowlan committed 17ec849e on 11.x
    Issue #3449743 by catch: Try to optimize test ordering when run-tests.sh...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.