[CI] Performance pipeline execution drops warmed caches

I mentioned this in this slack thread https://drupal.slack.com/archives/C51GNJG91/p1763133914571249?thread_ts=...

Just thinking out loud:
- When reading “cached”, read them from a “cached” folder.
- Make the last job in core pipeline (https://docs.gitlab.com/ci/yaml/#stage-post) update the “cached” folder with the newly generated artifacts

That way job will be getting the “cached” version even if a new one is being regenerated

eg (totally untested):

.cspell-cache: &cspell-cache
  # Fetch the cspell cache from the artifacts of the latest successful job from
  # the target branch. Allow the job to proceed and pass if the file doesn't
  # exist.
  - 'curl --location --output core/.cspellcache "https://git.drupalcode.org/api/v4/projects/{$CORE_GITLAB_PROJECT_ID}/jobs/artifacts/{$CACHE_TARGET}/raw/cached/core/.cspellcache?job=Lint%20cache%20warming" || true'

...

# This job should be real quick and run at the very end of the pipeline.
.post
  script:
    - cp -R $CI_PROJECT_DIR/core $CI_PROJECT_DIR/cached/core
  artifacts:
    paths:
      - cached/...

This was postponed but was brought in a slack thread. I created an MR with my thoughts from #4 but I'm not sure it can be tested unless merged because of the way it is architected.

This looks both clever and minimalist, although I'm not sure whether the gitlab bug is that the artifacts aren't available while the job is running, or whether they aren't available while the entire pipeline is running.

So as expected the Write lint caches job is super quick - see https://git.drupalcode.org/issue/drupal-3549110/-/jobs/7318879 - especially compared to the Lint cache warming job https://git.drupalcode.org/issue/drupal-3549110/-/jobs/7318861

Let's do it. Pipeline changes are readable and it should help a lot hopefully.

Reviewed and discussed the latest MR changes with @alexpott in slack, just re-confirming the RTBC this should hopefully make future changes more testable.

Two questions on the MR.

Nice review @longwave - addressed both of your comments. As these were small tweaks - setting back to RTBC. Now we get the full URL for all cache downloads by using the -v option on curl.

Hmmm... we've not solved the issue in the issue summary... need to have a think.

Okay I think this is ready. It no addresses the issue in the issue summary, is easier to test.

@longwave it still uses the .post job - I know you asked for a specific stage for that but I'm not sure that we want to do that. Can we do that in a follow-up if necessary?

Given .post is a built-in gitlab feature I think it's OK to expose it (even though I agree it doesn't look great) and then open a follow-up to explore changing to an explicit stage. Would be good to tweak things further from a known working state.

@alexpott also mentioned in slack that we should maybe look at ditching using the artifacts system for this and add an s3 bucket instead, which would require an infra change but could be worth exploring and would allow us to have only a single job again.

Actually now that #3387117: Enable distributed caching in GitLab Runner is fixed we should change the approach...

@catch that would be preferable tbh, but when we started this we couldn't hang on that since they were overbooked. Perhaps things are a little less stressfull right now (but i doubt it)

Can't do that because caches are not shared across projects... back to the commit @catch rtbc'd

Committed and pushed 2280c4a3f79 to 11.x and 3157cc68c87 to 11.3.x. Thanks!

Yeah un related... fixing...

Follow-up to make things even better #3558592: Improve lint cache warming even more