Problem/Motivation

Daily and weekly pipelines download lint caches. This is fine until the cache is more than 7 days old, then MR pipelines start to do a full phpstan scan, and we need to wait up to 24 hours for the daily pipeline to run now that on-commit jobs are switched off for development branches.

I think we can just skip downloading the artifact cache on those pipelines, this will force the cache to be produced from scratch each time, which will also mean less cache drift as things get committed - e.g. a maximum of one day drift instead of one week, which should in turn mean MR pipeline runs have less files to scan.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3620707

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

catch created an issue. See original summary.

catch’s picture

Status: Active » Needs review

catch’s picture

Added some changes to make sure all lint jobs actually run on daily tests, we were previously relying on branch pushes to warm the caches always. Note various jobs not running on the main daily pipeline: https://git.drupalcode.org/project/drupal/-/pipelines/945606

Even though the lint cache job is removed here, it doesn't stop the jobs from using the lint caches because we also download those in the jobs themselves. So we might need to make that conditional too.

catch’s picture

OK I think I got it working after struggling with basic YAML syntax. Now the daily jobs won't try to download the cache at all, so we should get a full run once per day.

catch’s picture

Issue tags: +Test suite performance
daffie’s picture

Status: Needs review » Needs work

All changes look good to me. Just a single question on MR.

Edit: disclamer, I have used AI on the review of the MR.

catch’s picture

Status: Needs work » Needs review

That job doesn't run very often but I think it's worth doing for consistency (and when it does run) - updated the MR.

catch’s picture

Errr I got confused which MR/issue I was working on, #8 is just wrong and reverted the commit.

The twig cs fixer job actually doesn't run on scheduled pipelines already - it only runs on commit, MR pushes or manually.

daffie’s picture

Status: Needs review » Reviewed & tested by the community

Looks good to me.
For me it is RTBC.

amateescu’s picture

Status: Reviewed & tested by the community » Needs review

Posted a few questions and suggestions in the MR.

catch’s picture

Status: Needs review » Reviewed & tested by the community

I think I got to all of @amateescu's review points one way or the other so moving this back to RTBC.

  • amateescu committed dd759f81 on 12.0.x
    task: #3620707 Skip the download caches job on weekly/daily pipelines...

  • amateescu committed 82659f80 on main
    task: #3620707 Skip the download caches job on weekly/daily pipelines...
amateescu’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed 82659f807bf to main and dd759f817d6 to 12.0.x. Thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

catch’s picture

Did some manual testing of this:

- rebased #3624454: Reduce CPU request for linting jobs to kick off a new pipeline that definitely only has the yaml changes vs HEAD, 5945 files were re-analyzed by phpstan.

- kicked off a daily main pipeline to theoretically refresh lint caches

- kicked off another pipeline for that issue, 0 files analysed https://git.drupalcode.org/project/drupal/-/jobs/12439407

So that should mean we only get drift between the lint caches and HEAD for up to 24 hours instead of up to 8 days.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

alexpott’s picture

Status: Closed (fixed) » Patch (to be ported)

The logic applies to 11.x and 11.5.x so we should backport this there too.

alexpott’s picture

Status: Patch (to be ported) » Fixed

Actually this doesn't need to be backported - see #3631912: Fix spell checking only on main and the write lint caches job

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.