When running tests from the command line, I currently have two options:
a) I can pass --verbose, in which case I get a really long list of mostly irrelevant passing tests that I have to scroll back through to check if anything failed; or
b) I can not pass --verbose, in which case if anything fails I need to re-run the test to find out what it was.

The vast majority of the time when I'm running a test I just want to know that everything passed, or if not then what I need to do to fix it. I propose listing all failed tests and exceptions by default, and adding a --summary-only option to support the current behaviour.

I wonder if this will confuse the d.o test infrastructure?

Files: 
CommentFileSizeAuthor
#1 drupal-2200747-run-tests-show-errors-1.patch2.46 KBianthomas_uk
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,338 pass(es). View
#1 drupal-2200747-run-tests-show-errors-1-will-fail.patch3.13 KBianthomas_uk
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,404 pass(es), 1 fail(s), and 0 exception(s). View

Comments

ianthomas_uk’s picture

FileSize
3.13 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,404 pass(es), 1 fail(s), and 0 exception(s). View
2.46 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,338 pass(es). View

Reroll, plus a version with a deliberate mistake to show this doesn't break d.o's test infrastructure (it shouldn't, as it doesn't change the behaviour of the --verbose flag)

sun’s picture

I've read the issue summary at least twice in the meantime, but I still do not understand what the problem and goal is... :-/ Can you clarify the summary?

The last submitted patch, 1: drupal-2200747-run-tests-show-errors-1-will-fail.patch, failed testing.

ianthomas_uk’s picture

Basically I don't find the current default output useful if a test fails, but the verbose output can get very long so isn't easy to read and doesn't give you any extra useful information if all tests pass.

This is an attempt to find a compromise between the two. It's still easy to read, but if a test fails it will tell you what (which is often enough, e.g. if it's a failure you've seen before, or is a test you were working on).

sun’s picture

Title: Make runtests.sh list failing tests by default » Make run-tests.sh show verbose output for failing assertions only
Issue tags: +Testing system, +DX (Developer Experience)

Note that we cannot change the default output of run-tests.sh, because that would affect the regular Review log of PIFR and testbots, too:

https://qa.drupal.org/pifr/test/748038#footer (expand the "Review log" fieldset)

The review log is big enough already now — it has to be transferred between the testbot and qa.drupal.org (in addition to the assertion result rows).

In other words, this will need a new flag/argument to run-tests.sh.


Aside from that, we can happily rethink the CLI --verbose output. This flag/option is not used by PIFR/testbots.

We might want to take inspiration from PHPUnit's output in case of a test failure. Our current ASCII table output is certainly very limiting.

That said, given how our (web) tests are designed, it is often helpful to see some level of context before a failing assertion (e.g., 3-6 pass/info messages), so as to better understand what happened before a failing assertion.

ianthomas_uk’s picture

Ah, ok. I thought the testbots were already using --verbose, because they get the assertion-level detail. If they are getting that by other means and log size is an issue then do we even want to be logging the test summary, which runs to several pages?

this will need a new flag/argument to run-tests.sh

I'd say the default behaviour should be the one most useful for humans. If test bot wants a different output, then it should be fairly easy to get testbot to pass the extra flag(s).

I didn't want to reduce the verbose output in case the full log is sometimes useful, as there would be no way of getting that. Yes, a few messages before the failure are often useful, but I didn't want to make the default output too noisy. I'm not particularly familiar with PHPUnit's output, so I'm not sure what we'd be getting inspiration from.

cilefen’s picture

What about --verbose (would be verbose on failures only) and in addition --debug, which would be verbose also on passing tests?

texas-bronius’s picture

I have been wanting this for some time, too: Running tests from command line in TDD, it's useful to see "All green!" or any exceptions and errors, but it is burdensome to see a sea of green and have to scroll back to see *whether* anything failed. Maybe if modifying --verbose is too risky, could we consider introducing a flag that means "--errors-and-exceptions" so like "--only-issues" or "--show-issues" might work?

Another idea which I think would also work is a way, from command line, to go back and list previous test outcomes. I guess this would involve storage somewhere: run tests as non-verbose, see a failure, then issue a followup command "show me the last test results details" which would then list either only failures or all, as if it were just run but with --verbose.

-Bronius

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.