When I submit a patch to some project hosted at Drupal.org, there is usually a robot that after just a few minutes tells that:

The last submitted patch, … failed testing.

For the record: I find this robot's behavior rather demotivating. How is it that it is so sure about my patch failing?

As an example, take the patch submitted in comment #2 in #2811273: 7.x-2.x test need update for compatibility with new permissions in core 7.50. It makes a tiny change to the project's README.txt. To be honest: This patch was just engineered to demonstrate that the automated testing framework frequently puts the blame in the wrong place.

I am a big fan of regression testing, but I must admit I struggle with automated testing as it is implemented at Drupal.org. The output is usually hard to decipher, but it seems that it always blame "the last submitted patch", no matter what the real problem is. I find that UX both confusing and demotivating.

Suggested solution: Keep track of the testing state of the code in the repo, and if that state already is "failed testing", and the patch fails to repair that, change the message to:

The last submitted patch, … did not succeed in repairing this project.

Only output the response:

The last submitted patch, … failed testing.

if the code in the repo has no testing failures, but automated tests fails after the patch is applied.

Comments

gisle created an issue. See original summary.

gisle’s picture

Issue summary: View changes

Added suggested solution.

gisle’s picture

Issue summary: View changes

Improved language.

drumm’s picture

Keep track of the testing state of the code in the repo

We do something like this already. If the branch test result isn’t a success, issue testing will wait for the branch to succeed. This can be overridden to go ahead and test the patch, to see if the patch improves the results.

In this case, the branch test results were almost a year old and were outdated. In more-active projects, this wouldn’t be a problem. #2651208: More-flexible, while reasonable, configured tests will provide more screen real-estate for setting up testing triggers for branches.

drumm’s picture

Status: Active » Closed (outdated)