Problem/Motivation
Daily and weekly pipelines download lint caches. This is fine until the cache is more than 7 days old, then MR pipelines start to do a full phpstan scan, and we need to wait up to 24 hours for the daily pipeline to run now that on-commit jobs are switched off for development branches.
I think we can just skip downloading the artifact cache on those pipelines, this will force the cache to be produced from scratch each time, which will also mean less cache drift as things get committed - e.g. a maximum of one day drift instead of one week, which should in turn mean MR pipeline runs have less files to scan.
Steps to reproduce
Proposed resolution
Remaining tasks
User interface changes
Introduced terminology
API changes
Data model changes
Release notes snippet
Issue fork drupal-3620707
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
catchComment #4
catchAdded some changes to make sure all lint jobs actually run on daily tests, we were previously relying on branch pushes to warm the caches always. Note various jobs not running on the main daily pipeline: https://git.drupalcode.org/project/drupal/-/pipelines/945606
Even though the lint cache job is removed here, it doesn't stop the jobs from using the lint caches because we also download those in the jobs themselves. So we might need to make that conditional too.
Comment #5
catchOK I think I got it working after struggling with basic YAML syntax. Now the daily jobs won't try to download the cache at all, so we should get a full run once per day.
Comment #6
catchComment #7
daffie commentedAll changes look good to me. Just a single question on MR.
Edit: disclamer, I have used AI on the review of the MR.
Comment #8
catchThat job doesn't run very often but I think it's worth doing for consistency (and when it does run) - updated the MR.
Comment #9
catchErrr I got confused which MR/issue I was working on, #8 is just wrong and reverted the commit.
The twig cs fixer job actually doesn't run on scheduled pipelines already - it only runs on commit, MR pushes or manually.
Comment #10
daffie commentedLooks good to me.
For me it is RTBC.
Comment #11
amateescu commentedPosted a few questions and suggestions in the MR.
Comment #12
catchI think I got to all of @amateescu's review points one way or the other so moving this back to RTBC.
Comment #15
amateescu commentedCommitted and pushed 82659f807bf to main and dd759f817d6 to 12.0.x. Thanks!
Comment #17
catchDid some manual testing of this:
- rebased #3624454: Reduce CPU request for linting jobs to kick off a new pipeline that definitely only has the yaml changes vs HEAD, 5945 files were re-analyzed by phpstan.
- kicked off a daily main pipeline to theoretically refresh lint caches
- kicked off another pipeline for that issue, 0 files analysed https://git.drupalcode.org/project/drupal/-/jobs/12439407
So that should mean we only get drift between the lint caches and HEAD for up to 24 hours instead of up to 8 days.
Comment #19
alexpottThe logic applies to 11.x and 11.5.x so we should backport this there too.
Comment #20
alexpottActually this doesn't need to be backported - see #3631912: Fix spell checking only on main and the write lint caches job