Patch having test with "PHP Fatal error" is marked as PASSED

So the issue here is that both simpletest and phpunit create a 'fake failed test', and then run the test. If there is a fatal error when it runs the test, we fall back on the fake failed test.

The problem, here, is that the fatal error is happening during the creation of the fake failed tes (here http://cgit.drupalcode.org/drupal/tree/core/modules/simpletest/src/TestB...) t - it uses reflection to get some information about the test its about to run, and thats when the fatal error happens, so the fake fail never happens and the test itself seems invisible to the test runner:

We're adding a jenkins plugin to scan the output of the tests to verify that this sort of error is caught.

Ugh. Dangit. https://dispatcher.drupalci.org/job/default/21328/ Jenkins marks the build unstable, but we'll have to do some more work on the d.o. side to make it concern itself with jenkins specific type of results there.

Now that legacy tests are off, this is ultra-critical.

This needs a fix on the drupal.org side at this point: http://cgit.drupalcode.org/project_issue_file_test/tree/includes/pift_ci... is checking # of test failures, when it should *also* be checking on whether or not the build is stable.

Just hit this again. Had a patch with dozens of fatal errors, patch "passed". Thankfully I thought to check...
Is there a higher priority than "critical"? :)

Trying to implement a higher priority.

It is possible to fix this in core itself:

+     $caller = array(
+       'file' => __FILE__,
+        'line' => __LINE__,
+        'function' => $class . '->' . $method . '()',
+     );
+      $test_completion_check_id = TestBase::insertAssert($this->testId, $class, FALSE, 'The test did not complete due to a fatal error.', 'Completion check', $caller);

      // Insert a fail record. This will be deleted on completion to ensure
      // that testing completed.
      $method_info = new \ReflectionMethod($class, $method);
      $caller = array(
        'file' => $method_info->getFileName(),
        'line' => $method_info->getStartLine(),
        'function' => $class . '->' . $method . '()',
      );

+      TestBase::deleteAssert($test_completion_check_id);
      $test_completion_check_id = TestBase::insertAssert($this->testId, $class, FALSE, 'The test did not complete due to a fatal error.', 'Completion check', $caller);

It is ugly but would work ...

The code is less ugly than the bug it fixes - I think that's worth a core issue and we should be able to verify it.

I'm finding that the problem happens way earlier and it occurs in the autoloader so we have no way of catching it :(

I've hacked run-tests.sh to only run a broken test (because it fails even earlier if you use --class) and here's the backtrace...

Drupal test run
---------------

Tests to be run:
  - Drupal\field\Tests\EntityReference\EntityReferenceSettingsTest

Test run started:
  Thursday, November 12, 2015 - 14:32

Test summary
------------

FATAL Drupal\field\Tests\EntityReference\EntityReferenceSettingsTest: test runner returned a non-zero error code (255).
PHP Fatal error:  Trait 'Drupal\field\Tests\EntityReference\EntityReferenceTestTraits' not found in /Volumes/devdisk/dev/sites/drupal8alt.dev/core/modules/field/src/Tests/EntityReference/EntityReferenceSettingsTest.php on line 23
PHP Stack trace:
PHP   1. {main}() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:0
PHP   2. simpletest_script_execute_batch() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:90
PHP   3. simpletest_script_cleanup() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:578
PHP   4. is_subclass_of() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:730
PHP   5. spl_autoload_call() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:730
PHP   6. Symfony\Component\ClassLoader\ApcClassLoader->loadClass() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:0
PHP   7. require() /Volumes/devdisk/dev/sites/drupal8alt.dev/vendor/symfony/class-loader/ApcClassLoader.php:110

Fatal error: Trait 'Drupal\field\Tests\EntityReference\EntityReferenceTestTraits' not found in /Volumes/devdisk/dev/sites/drupal8alt.dev/core/modules/field/src/Tests/EntityReference/EntityReferenceSettingsTest.php on line 23

Call Stack:
    0.0052     446264   1. {main}() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:0
    0.9520   17781584   2. simpletest_script_execute_batch() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:90
    1.9786   17895920   3. simpletest_script_cleanup() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:578
    1.9786   17896016   4. is_subclass_of() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:730
    1.9786   17896456   5. spl_autoload_call() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:730
    1.9786   17896536   6. Symfony\Component\ClassLoader\ApcClassLoader->loadClass() /Volumes/devdisk/dev/sites/drupal8alt.dev/core/scripts/run-tests.sh:0
    1.9794   17933952   7. require('/Volumes/devdisk/dev/sites/drupal8alt.dev/core/modules/field/src/Tests/EntityReference/EntityReferenceSettingsTest.php') /Volumes/devdisk/dev/sites/drupal8alt.dev/vendor/symfony/class-loader/ApcClassLoader.php:110

lol

Yay I can make it fail :)

Seems we have a bit of a VW Bug here...

:P

The fix is a bit ugly, but the bug is considerably uglier and nice to be able to fix this directly in core considering we're already trying to handle it but unsuccessfully.

Restoring the "merely critical" priority since that's sufficient for core. ;)

Committed and pushed to 8.0.x. Thanks!

Per #29.

Great, thank you. This is definitely a better approach than trying to scour the output looking for the word Fatal (which had a nasty tendency to also be in commit messages that show up in the logs as well).