Allow known flaky tests to be automatically repeated

If a test is that flaky (and I personally have seen multiple failures on RTBC patches for that QuickEditIntegrationTest), I think the best thing to do would be to remove it from the code base as soon as it's identified as a major annoyance, and then have an issue that adds it back in once it has been fixed.

This is what I have in my project:

<?php

namespace Drupal\Tests\my_project_tests\Traits;

trait RetryTrait {

  public function runBare() {
    $e = NULL;
    $numberOfRetries = $this->getNumberOfRetries();
    if (FALSE == is_numeric($numberOfRetries)) {
      throw new \LogicException(sprintf('The $numberOfRetries must be a number but got "%s"', var_export($numberOfRetries, TRUE)));
    }
    $numberOfRetries = (int) $numberOfRetries;
    if ($numberOfRetries <= 0) {
      throw new \LogicException(sprintf('The $numberOfRetries must be a positive number greater than 0 but got "%s".', $numberOfRetries));
    }
    for ($i = 0; $i < $numberOfRetries; ++$i) {
      try {
        parent::runBare();
        return;
      } catch (\PHPUnit_Framework_IncompleteTestError $e) {
        throw $e;
      } catch (\PHPUnit_Framework_SkippedTestError $e) {
        throw $e;
      } catch (\Throwable $e) {
        // last one thrown below
      } catch (\Exception $e) {
        // last one thrown below
      }
    }
    if ($e) {
      throw $e;
    }
  }

  /**
   * @return int
   */
  private function getNumberOfRetries() {
    $annotations = $this->getAnnotations();
    if (isset($annotations['method']['retry'][0])) {
      return $annotations['method']['retry'][0];
    }
    if (isset($annotations['class']['retry'][0])) {
      return $annotations['class']['retry'][0];
    }
    if (isset($this->retry) && (!empty($this->retry))) {
      return $this->retry;
    }
    return 1;
  }
}

Steps to reproduce

Mark a patch RTBC and wait a few days/weeks for a random test failure.

Delightfully lovely :) and so true.

+1 on the idea of this patch.

This seems pragmatic. I think there should always be an issue open to remove it and it should be a matter of last resort. But the random fails in the JS tests have prove intractable for some reason.

With issues like #3037436: [random test failure] Make QuickEditIntegrationTest more robust and fail proof I think we should have been marking those as skipped (leaving the code there) with issues to diagnose and re-enable them, but there's a question as to why we haven't done that, so we should definitely at least discuss this option instead.

In PHPUnit upstream, this is being proposed in #[Retry] attribute to support retrying flaky test #6182

I suggest to continue in #3564900: Let run-tests.sh re-run flaky test classes on failure as now the overall context changed, and in fact the situation deteriorated quite a lot.