Problem/Motivation

PHPUnit Build and Nightwatch jobs are failing very often. Almost every MR I push to fails at least once from one of these jobs. We have previously tried to fix the PHPUnit Build random fails in #3497701: [random test failure] PackageUpdateTest::testPackageUpdate but they still happen very often.

Proposed resolution

Add retry: 1 to these jobs.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3539923

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

acbramley created an issue. See original summary.

acbramley opened merge request !12922

acbramley’s picture

acbramley commented
Status: Active » Needs review
cmlara’s picture

cmlara
he/him
commented
Related issues: +#3165263: Allow known flaky tests to be automatically repeated

Linking a related, and still open, issue that also discusses retrying known flakey tests. No decision is in this issue, and the issue predates GitLabCI.

mondrake’s picture

mondrake
Location 🇮🇹
commented
Status: Needs review » Reviewed & tested by the community

I think this makes sense for now. It’s a small change, if it does not work it can be easily reverted.

In PHPUnit upstream, this is being proposed in #[Retry] attribute to support retrying flaky test #6182. That will allow working on a test class instead of the entire CI job, but surely will only happen in PHPUnit 12 or 13.

catch’s picture

catch
he/him
Primary language English
commented
Related issues: +#3467492: [policy, no patch] Replace Nightwatch with Playwright, +#3534109: Permanently maintain a sandbox directory in package_manager

Linking related issues. I think a single retry is worth doing for these two jobs since we know they're very flaky. Adding related issues that are potentially why they're flaky. We're most likely to fix nightwatch but removing it entirely, and in that case the job will go. For build tests we should try to make the tests less flaky and remove the retry option again.

quietone’s picture

quietone commented

I support this change. It will simply do what I do manually to get those tests to pass.

  • catch committed 4d82ebf8 on 11.2.x 
    Issue #3539923 by acbramley, mondrake: Allow PHPUnit build and...

  • catch committed cc436a7b on 11.x 
    Issue #3539923 by acbramley, mondrake: Allow PHPUnit build and...
catch’s picture

catch
he/him
Primary language English
commented
Status: Reviewed & tested by the community » Fixed

Yep let's do it - we can always revert if we think we've made them generally more reliable, or use #[Retry] when it's available.

Now that this issue is closed, please review the contribution record.

As a contributor, attribute any organization helped you, or if you volunteered your own time.

Maintainers, please credit people who helped resolve this issue.

catch closed merge request !12922

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.