Stabilize OpenTelemetryAuthenticatedPerformanceTest

OK that either means there's still random failures or it means it was 'randomly passing' before.

I think this is the only performance test in core that covers contextual links, which are rendered via a custom AJAX request, which will be even more delayed than asset/image requests - so it could be that.

Pushed a commit that tries to disable local storage during performance tests - this should mean we don't get a situation where one test run happens to have something in local storage and another doesn't. I am not entirely sure if the syntax is correct but it passes locally and also passed a couple of test runs.

Kicked off a repeat test run here: https://git.drupalcode.org/issue/drupal-3554646/-/jobs/7067105 - and that didn't work, or at least not enough. However it's interesting that all the failing tests took more than 200s to run and everything 200s or less passed, so at least there's a pattern.

Tried uninstalling contextual module in the test, but we can't do that because layout builder depends on it. It would be possible to remove that dependency in performance_test module in a hook_system_info_alter() maybe, but would be more realistic to manually add a load of permissions to a user instead of using the admin role, and just don't give that user access to contextual links - although no guarantee that's 100% the problem but it feels like it's worth trying to rule out.

https://git.drupalcode.org/issue/drupal-3554646/-/jobs/7071428 looking better (although not finished yet).

Without the longer sleep after user login, a handful of tests would run for about double time and fail with slightly different numbers. Compensated for that with shorter sleeps for the actual page cache warming which appears to still work fine. Going to kick off another repeat class test job just for luck, but this seems to be it.

Would be nicer if we didn't have to have the sleeps here and could instead assert that every http request issued from the browser gets a response, but even that wouldn't help with post response tasks. Essentially several of the optimisations in core to improve response times, parallelise work etc. make it harder to write accurate performance tests, but of all the performance problems we have, a few sleep() in a few tests is not the biggest.

Fresh repeat test class job https://git.drupalcode.org/issue/drupal-3554646/-/jobs/7076879

https://git.drupalcode.org/issue/drupal-3554646/-/jobs/7076879 has some more failures, but one of the tests took more than 10 minutes to complete, so I'm wondering if that's partly an artifact of the repeat tests class job itself - e.g. normal functional test runs don't try to install Umami 100 times simultaneously. So it would be good to try this and we can open another issue if we see any more random failures.

This failed again on an MR almost immediately after it was committed, so more to do here.

Figured it out.

The cache warming requests for asset aggregates/image derivatives on the node page weren't exactly the same as the one we're recording cold cache backend performance for - because the test logs in a user first and that warms some different caches and assets prior to warming the node page. So there was at least one aggregate that was unique to the performance recording request, which causes an extra Drupal request and bootstrap etc.

The solution is to visit node/1 with an empty cache after logging in, then the same asset libraries are requested and the aggregates created.

https://git.drupalcode.org/issue/drupal-3554646/-/jobs/7142642 is 100 green runs of this test.

https://git.drupalcode.org/issue/drupal-3554646/-/jobs/7143359 is a second run I'm hoping will be green. edit: all green.

Answered the review comments on the MR.

Rebased.