Have PIFR provide a results summary even for failed tests. [#1615232]

Initial description (from #1565718: Script test runner does not clean after fatal error):

Looking a http://qa.drupal.org/pifr/test/279838, which has some fatal errors but also working tests. The testbot however now doesn't list anymore which tests failed and which don't, so you have to go through the detail log to find out what exactly happened. So the nice thing is that you actually can do that and actuall see a PHP fatal error there but there is no nice overview anymore.

Details:

Currently, if an error is encountered during a testing stage, then the error is logged and the routine breaks out before incrementing the $result['code'] counter. This occurs in the run() routine within client.inc.

This prevents the get_results() routine in pifr_simpletest.client.inc from generating a results summary due to the $result['code'] check at the top of that routine.

It should be feasible to extend the get_results() routine to add some logic for the $result['code'] == ($this->result_codes['fail'] - 1) scenario, to try and determine if any simpletest results actually exist for that test; and if so, attempt to generate a partial results summary for any tests which were run before the error condition was encountered during the 'review' operation.

Comment	File	Size	Author
#3	PIFR_return-partial-results-on-fatal_1615232-3.patch	3.35 KB	jthorson

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

sun

German

Karlsruhe

CreditAttribution: sun commented 4 June 2012 at 10:26

Technically, I'd expect the client to always retrieve and process any kind of results that have been written, regardless of success or error state.

Given the changes from #1565718: Script test runner does not clean after fatal error, for each test, there will always be at least the initially inserted failed assertion that checks for a fatal error (and which is only deleted after a successful test run).

In other words, I'd rather suggest to remove the conditional logic and code for "no results report upon code/error X" entirely. I don't know why that exists in the first place.

Comment #2

jthorson CreditAttribution: jthorson commented 4 June 2012 at 15:14

The conditional logic is intended to catch testbot and/or early test failures (such as failed checkouts, syntax errors, patches which don't apply, etc), as opposed to simpletest failures. The suggestion to use $result['code'] == ($this->result_codes['fail'] - 1) is because any result codes lower than this would indicate the bot never got around to calling run-tests.sh in the first place; and thus there will be no assertion results to process.

The other thing that would need to be considered developing the solution to this issue is that the presence of results does not guarantee that run-tests.sh was executed either; as failure during a re-test could occur before obsolete results from the previous run are cleared out.