Problem/Motivation

When a Panther PHPUnit run has errors or failures, the output is hard to read. Assertion failure messages include the full HTML page dump that Panther attaches to the exception, and there is no consolidated list of what failed at the end of the run, unlike the standard PHPUnit result summary. Two subscribers also record the wrong status: TestErroredSubscriber and TestMarkedIncompleteSubscriber both route their events through markTestFailed(), so errored and incomplete tests are reported as failures.

Steps to reproduce

  • Run a functional JavaScript test that fails an assertion. The failure message contains the entire rendered HTML page, and no numbered error/failure summary is printed at the end of the run.
  • Mark a test errored or incomplete, and observe it counted as a failure.

Proposed resolution

  • Print a PHPUnit-style end-of-run summary listing errors, failures and warnings as a numbered list, each with a clean single-line message plus its stack trace, followed by a final test count line (OK (N tests) or FAILURES! Tests: …).
  • Store the error message and stack trace separately on TestInformation instead of concatenating them, and strip the HTML page dump down to the first line.
  • Fix TestErroredSubscriber and TestMarkedIncompleteSubscriber to record the correct status via markTestErrored() and markTestIncomplete().

Remaining tasks

Remaining tasks

  • Review the merge request.

User interface changes

PHPUnit test output now ends with a numbered summary of errors, failures and warnings plus a test count line, replacing the inline HTML page dump in failure messages.

API changes

TestContext::markTestErrored(), markTestFailed() and markTestInconal <code>string $stackTrace parameter. FormatterInterface gainsresultSummaryHeader(), resultSummaryItem() and testCountSummary(). The renderMessage() helper in MessageTrait is removed in favour of separate getFirstLine() and renderStackTrace() calls.

Data model changes

TestInformation gains a string $stackTrace property.

Comments

lussoluca created an issue. See original summary.

lussoluca’s picture

Issue summary: View changes
lussoluca’s picture

Title: Add PHPUnit result summary and fix navigation race » Add PHPUnit result summary

  • lussoluca committed 50aea2c9 on 3608767-phpunit-result-summary
    fix(tests): use Crawler::text() instead of undefined getText()
    
    Refs:...
lussoluca’s picture

Assigned: lussoluca » Unassigned
Status: Active » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.