drush -test-run should exit with failure if tests fail [#1802948]

I think that running:

drush test-run --all

Should return a non-zero exit code when a test fails that it's testing. I know that the Drupal core run-tests.sh doesn't either, but I still think it's poor form here.

Tools like travis or Jenkins use the exit codes to determine if the test was a success/fail, so at the moment could always think things are working just fine.

Comment	File	Size	Author
#2	drush-1802948-non-zero-test-exit.patch	425 bytes	Steven Jones
#1	drush-1802948-non-zero-test-exit.patch	407 bytes	Steven Jones

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

File	Size
drush-1802948-non-zero-test-exit.patch	407 bytes

Comment #2

Steven Jones CreditAttribution: Steven Jones commented 4 October 2012 at 08:56

File	Size
drush-1802948-non-zero-test-exit.patch	425 bytes

Actually I think that returns the wrong way around.

Comment #3

moshe weitzman CreditAttribution: moshe weitzman commented 5 October 2012 at 03:01

Assigned:

Unassigned

» msonnabaum

Not sure. if you are running with Jenkins, you really shoud use the --xml option which better communicates failures, timing, etc. Any opinions here?

Comment #4

greg.1.anderson CreditAttribution: greg.1.anderson commented 5 October 2012 at 16:23

I think that it is most appropriate to return the correct status result code as in #2, even if the same information also appears inside the output produced via the --xml option.

Comment #5

moshe weitzman CreditAttribution: moshe weitzman commented 5 October 2012 at 19:05

With that change, you can't distinguish between a simplet test failure versus test-run itself failing.

Comment #6

greg.1.anderson CreditAttribution: greg.1.anderson commented 5 October 2012 at 20:16

Ideally, I would expect that both of those failures would return an error status result, with the more specific error information being included in the output generated with the --xml option (or possibly introduce different status return codes for Drush -- but we've never supported that before). I won't push for a change if the current implementation works better with our existing test harness, though.

Comment #7

alberto56 CreditAttribution: alberto56 commented 26 July 2013 at 19:10

Version:

» 8.x-6.x-dev

you can't distinguish between a simpletest failure versus test-run itself failing.

Exit code 1 will cause the Continuous integration tool to alert the developers that action is required, and we can then dig further to see if it is test-run, or the simpletest tests, or something else.

When the simpletest fails, are there any situations where a normal exit code is required?

(Posting a comment required me to add a version -- I added the latest available)