Problem/Motivation

with-composer jobs have 'needs' for phpcs and phpstan, but as well as /vendor this also downloads the phpstan and phpcs caches, which we don't want.

See https://docs.gitlab.com/ci/yaml/#needsartifacts

Steps to reproduce

Proposed resolution

To avoid downloading massive caches to 15+ different test jobs, always run the composer job and get artifacts only from there.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3620911

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

catch created an issue. See original summary.

catch opened merge request !17002

catch’s picture

catch
he/him
Primary language English
commented
Issue summary: View changes
catch’s picture

catch
he/him
Primary language English
commented
Issue tags: +Test suite performance
catch’s picture

catch
he/him
Primary language English
commented
Title: Do less downloading of artificats » Do less downloading of artifacts
catch’s picture

catch
he/him
Primary language English
commented
Status: Active » Needs review

OK well this passes tests, it's not slower, cannot tell if it's faster really. The artifact downloading happens within the container build step which is a bit of a black box.

catch’s picture

catch
he/him
Primary language English
commented
Title: Do less downloading of artifacts » Only download artifacts we actually need during CI pipelines
smustgrave’s picture

smustgrave commented
Status: Needs review » Reviewed & tested by the community

Makes sense if I'm understanding that right at least to pull from the composer job. Seems like one of those lets try out for a bit and see. Going to mark it.

catch’s picture

catch
he/him
Primary language English
commented

Rebased after #3631331: [ci] Cache Composer vendor directory. Now we don't need to run the composer job, but we can still avoid downloading the artifacts from the lint jobs.

longwave’s picture

longwave
he/him
Primary language English
Location UK
commented
Status: Reviewed & tested by the community » Needs review

The Composer job uploads vendor:

https://git.drupalcode.org/project/drupal/-/jobs/12926925

Uploading artifacts...
vendor/: found 9997 matching artifact files and directories 
composer.lock: found 1 matching artifact files and directories 
composer/Metapackage/: found 8 matching artifact files and directories 
Uploading artifacts as "archive" to coordinator... 201 Created  correlation_id=1212768540 id=12926925 status=201 token=69_YCwU_n

The PHPCS job also uploads vendor:

vendor/: found 9997 matching artifact files and directories 
composer.lock: found 1 matching artifact files and directories 
composer/Metapackage/: found 8 matching artifact files and directories 
test-artifacts/phpcs: found 2 matching artifact files and directories 
core/.phpcscache: found 1 matching artifact files and directories 
Uploading artifacts as "archive" to coordinator... 201 Created  correlation_id=3391016975 id=12926928 status=201 token=69_o_ekJq

This only takes a couple of seconds but it seems a waste of time both jobs doing it?

longwave’s picture

longwave
he/him
Primary language English
Location UK
commented

Also given that PHPCS takes about 3-4 seconds to run and PHPStan also doesn't take very long, maybe we should just combine all these into one job?

catch’s picture

catch
he/him
Primary language English
commented

This only takes a couple of seconds but it seems a waste of time both jobs doing it?

That's a good point dropped it from phpcs.

Also given that PHPCS takes about 3-4 seconds to run and PHPStan also doesn't take very long, maybe we should just combine all these into one job?

This is mostly true except when they have a lot of files, also those are the only two jobs that block tests from running, so I think it's OK for them to have their 3 CPUs between them and run concurrently for now. Could be worth an issue to try though - always a balance between job time and pod waiting time.

smustgrave’s picture

smustgrave commented
Status: Needs review » Reviewed & tested by the community

Believe feedback for this one was answered. Hope I’m right