PHP Fatal Errors during unit tests should show as a failure [#2538478]

https://dispatcher.drupalci.org/job/default/1635/console is an example of where a fatal error happened during a unit test, but did *not* show a fail during that test.

We need to investigate whether this is happening on the old bots as well, or if this is a new bug.

Comments

Comment #1

Mixologic

He/Him/His

English

Portland, OR

commented 10 August 2015 at 23:01

This does not happen on the old bots. they fail. The new bots do not, and there are plentymany examples.

Comment #2

webchick

she/they

English

Vancouver 🇨🇦

commented 11 August 2015 at 18:02

Priority:

Normal

» Critical

That seems pretty critical to address.

Comment #3

fabianx commented 20 August 2015 at 16:31

Agree, that is a hard blocker for shutting down the old bots.

Comment #4

mradcliffe

he/him

English

commented 20 August 2015 at 16:49

Is this due to Jenkins not failing the build because there are no "fails" but there are "exceptions"?

Or is this due to a BuildStep not setting errorStatus on the job on an exception?

I looked at \DrupalCI\Plugin\BuildSteps\publish\JunitXMLFormat class, and this does categorize exceptions directly from the simpletest table. A fatal error is listed as an 'exception' not a 'fail'. Does Jenkins only fail builds if it has a fail count > 0? I can't find out any of this information on the Jenkins documentation pages.

Comment #5

jthorson commented 20 August 2015 at 17:05

PIFR parses the run-tests.sh output for the "PHP Fatal" pattern ... currently, DrupalCI does not. This is essentially a regression which hasn't been built into DrupalCI yet.

I don't believe that the tests which result in a PHP Fatal in the output necessarily complete to the point where the exception would be reported in the simpletest result tables.

Comment #6

jthorson commented 20 August 2015 at 17:07

Without taking a deep look into the code, my suggested resolution would be to tee the output into a file while the run-tests.sh command is running, and then have a post-execute build step (prior to the JunitXML parsing) which does some validation on this output file and fails if it detects any PHP Fatal error messages in the run output.

Comment #7

Mixologic

He/Him/His

English

Portland, OR

commented 28 August 2015 at 19:58

I noticed that not all fatal errors were being handled like this, and that some of them actually report as a true failure. (For example: https://dispatcher.drupalci.org/job/default/9727/testReport/Plugin/Drupa...)

So I dug a little deeper and it turns out its a problem with how we are handling phpunit tests vs. simpletest tests. Simpletest assumes the test is a failure from a fatal error unless otherwise told it is passing: http://cgit.drupalcode.org/drupal/tree/core/modules/simpletest/src/TestB...

When phpunit fatal errors, it creates a 0 byte xml file in /sites/default/files/simpletest/phpunit-xx.xml. When it comes time to parse that output (http://cgit.drupalcode.org/drupal/tree/core/modules/simpletest/simpletes...), In line 673 it checks and if the file is 0- bytes, it simply bails out.

What I propose is that we create a default junit xml formatted 'failure' file in simpletest_phpunit_run_command().(http://cgit.drupalcode.org/drupal/tree/core/modules/simpletest/simpletes...) that would then get overwritten on a successful execution of that particular test run.

The alternative is a pretty sad hack that involves parsing the run-tests.sh output to see if the word fail is there, which doesnt provide developers with information as to *where* a fatal error occurred without digging into the test suite. Even then, it just displays the error, but we dont know which class was attempting to be tested.

This falls into the same category of problems where in order to fix drupalci, we need to fix core's testrunning tools, and its not ultimately clear the best course of action going forward. (e.g. https://www.drupal.org/node/2551981)

Comment #8

Mixologic

He/Him/His

English

Portland, OR

commented 31 August 2015 at 20:49

Created #2560643: Fatal PHPUnit tests are not always reported as failure in core to get this fixed properly. In the meantime we'll work around it by doing the same "parse the logs and look for a Fatal error" that we're currently doing in pift.

Comment #9

dawehner

German

commented 9 September 2015 at 17:26

https://www.drupal.org/node/2546618#comment-10309439 is a slightly different problem, not sure whether its at all related to this issue.
This is a segfault while running the tests, and well, this was not found at all.

Comment #10

basic commented 10 September 2015 at 23:04

Status:

Active

» Postponed

This is now waiting on #2560643: Fatal PHPUnit tests are not always reported as failure to get fixed. Once this is in core this issue can be resolved.

Comment #11

Mixologic

He/Him/His

English

Portland, OR

commented 11 September 2015 at 22:12

Status:

Postponed

» Fixed

This is fixed now, #2560643: Fatal PHPUnit tests are not always reported as failure is in.

Comment #12

25 September 2015 at 22:14

Status:

Fixed

» Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

PHP Fatal Errors during unit tests should show as a failure

Comments

Comment #1

Comment #2

Comment #3

Comment #4

Comment #5

Comment #6

Comment #7

Comment #8

Comment #9

Comment #10

Comment #11

Comment #12

Parent issue

Referenced by

News items

Our community

Documentation

Drupal code base

Governance of community