Let run-tests.sh re-run flaky test classes on failure

Problem/Motivation

There are way too many test failures in CI. In fact, it's nearly impossible to have a fully green commit or daily pipeline.

This impairs the credibility of the framework in identifying real issues vs just reporting flakiness.

It's possible that at least some of the failure are due to concurrency. There is work upstream in #[Retry] attribute to support retrying flaky test #6182 to implement retries, but even if it will ever see the light, it will enable retries within the same PHPUnit process where the failure happened, which may not be 'enough' in run-tests.sh as the concurrency factor will still be in the equation during the retries.

Steps to reproduce

Look at any pipeline run and get frustrated by the number of failed jobs.

Proposed resolution

Introduce a configuration file for run-tests.sh, holding a list of tests that can be retried on failure, and the allowed # of retries.

During the concurrent test run, keep track of the failing tests.

Once the concurrent test run has completed, loop through the remaining failed tests and run them in non-concurrent mode (or lower concurrency, let's see).

If there are still failing tests after all the retries, fail the entire job.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet