[random test failures] Race condition in state when individual keys are set with an empty cache

Problem/Motivation

Various test modules use the state API to track things between the tested site and the test, after #2575105: Use cache collector for state this creates race conditions, in combination with #3295790: Post-response task running (destructable services) are actually blocking; add test coverage and warn for common misconfiguration - because the state cache item is written as a post response task.

The race condition is as follows:

1. drupalGet(), state cache miss.
2. Test class, calls State:set(), which calls invalidate, but there's nothing to invalidate yet
3. request from #1 is still running end of request tasks, we get to updateCache() in CacheCollector, there was no cache at the beginning of the request, and there's still no cache now, so it just writes to the cache.
4. Test class calls drupalGet(), and this now gets a cache without whatever happened in #2.

State has an explicit race condition prevention mechanism when a request begins with a cache item, writes to it at the end of the request, but it's been upated in-between. However, it doesn't handle the case when there is a cache miss at the beginning of the request and the cache is invalidated again before it tries to write.

This case would be extremely unlikely on a real site - you usually either have an empty cache after an invalidation, in which case it's safe to write, or you have a non-empty cache before an invalidation, in which case we already handle it, but the 'clean' environment of the test means we're reproducing it in a fraction of test runs.

We can prevent this by writing a cache item from State::set(), and in CacheCollector, not writing to the cache, if it was empty at the beginning of the request and has been written before the end of the request. Most CacheCollector implementations don't need this protection because they are caching discovery and similar - which can be invalidated, but not individually written to.

Steps to reproduce

Proposed resolution

For test randomness, either using TestWaitTerminate in these tests, or switching from state to key/value would prevent the random test failures.

However, it's also possible to add race condition protection to CacheCollector + state.MR 7336 attempts to do this and is the MR to review.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet