We've seen a couple error scenarios where no tests are actually run for a branch test or patch ... occasionally, this is due to run-tests.sh not being passed any tests via the --file argument. See the related PIFT issue #1156976: 0 tests passing is not green.
We should detect this condition and return an appropriate response instead of allowing a zero-test test to run and return green.
Comments
Comment #1
jthorson commentedPatch attached.
Comment #2
jthorson commentedWell ... that didn't work. New approach.
Comment #3
rfayRelated issue: #1156976: 0 tests passing is not green
Comment #4
jthorson commentedMerged this patch into the one posted at #1126112-11: Full tests of Drupal being done for some contrib patches, as they would otherwise conflict.
Comment #5
jthorson commentedFix committed in referenced patch.
Comment #7
jthorson commentedPatch reversed, as the 'fail' condition prevents the ability to leverage testbots to ensure a patch can be applied against a project with no tests.
Comment #8
jthorson commentedNew approach, which should catch 'No tests found' due to testbot errors, but will not execute run_tests.sh if no arguments are found for the '--file ' parameter.
Comment #9
jthorson commentedAfter more thought, this won't actually catch the 'no tests found' error ... but is still a step in the right direction.
Comment #10
jthorson commentedPosting patch here so I can validate on scratchtestbot ...
Comment #11
jthorson commentedThere ... I think this does the trick.
This patch takes the following approach:
i) Check for empty arguments on the "--file" parameter, and skip run_tests.sh if there are no tests to run
ii) Scan the review output for "PHP Fatal error", and fail the test if one is found ... thus fatal errors which interrupt the review process do not result in 'green' results.
This check could possibly be extended to include PHP Warning ... but I think the 'Fatal error' version this should catch 90% of our problem cases now.
Comment #12
jthorson commentedLooks promising ... tests result can be found at:
zero tests passed: http://rfay.redesign.devdrupal.org/node/1073590/testing-status
PHP Fatal error: http://rfay.redesign.devdrupal.org/node/19304/testing-status (6.x-1.x)
With actual tests passed: http://rfay.redesign.devdrupal.org/node/19304/testing-status (7.x-2.x)
Comment #13
jthorson commentedRerolled patch.
Comment #14
jthorson commentedCommitted to 6.x-2.x (264fb4f).
Couldn't find a patch which contained a FATAL but didn't break the testbot, so that component of the patch is unproven. Will need to watch for this scenario after deployment.
Comment #15.0
(not verified) commentedUpdated to refer to pift issue.