It could be useful to detect whether there are any PHP errors or notices being generated during Behat tests. A demonstration of the idea is here: https://www.godel.com.au/blog/use-behat-track-down-php-notices-they-take....

The suite could output a list of PHP notices at the end of the results, just before the summary of passed/failed scenarios. You could also do this after every scenario, but that might be too much overhead.

Patch to come.

Comments

cboyden’s picture

Status: Active » Needs review
StatusFileSize
new1.58 KB

Patch is attached. This code truncates the watchdog table at the beginning of the suite and prints out all PHP notices etc. at the end.

cboyden’s picture

There are a few options if you want PHP notices to have some effect on the exit status of automated tests.

On Travis, you could add a step after each suite that would check the watchdog. If PHP notices existed, you could set the exit status to error. Problems with this approach:

  • drush wd-list requires you to enter a number corresponding to the type of error you want to list. The number changes depending on what's been logged.
  • drush wd-show complains about "Unrecognized message type" if you check for php messages and there aren't any. You'd have to parse the response to see that there were no errors (if it returned nothing, that's easier to check for in the shell).
  • drush sqlq (to count the number of results) outputs a MySQL warning and echoes the selected column header before it outputs the useful information (the count) - would also require parsing the result in the shell.

You could also adapt the above patch to write a file with the PHP errors for each suite (or scenario, that'd be more useful). Travis could check for the existence of files at the end of the run_tests step and set the exit status to error if they existed. Problems with this approach:

  • Mostly for Travis: If you want the files to be available to look at after the test run is done and the environment is destroyed, you have to output them to the screen. That should be doable.
  • Mostly for local testing: If you want the results files to be saved outside the site root, you'll have to pass in a parameter for the location using behat.yml. Again, should be doable, but it's something new to figure out.
dsnopek’s picture

Here's what I'd personally think would be awesome:

  • Checking at the end of each scenario, to see if there are PHP notices. If so, output them to the screen without throwing an exception.
  • A configurable option in behat.yml (or an environment variable if that's saner to do), which takes a file to also write the PHP notices to. For simplicity, it could write all PHP notices to the same file, but put a little header before them saying the name of the scenario. This file should be deleted at the start of the suite, and only created and appended to if any PHP notices happen.
  • Then for Travis, we can check for the existance of this file and give a non-zero exist status if it exists.

This way, everyone always gets to at least see the notices, and we can still make it an error in Travis.

What do you think?

lsolesen’s picture

Status: Needs review » Needs work
cboyden’s picture

Status: Needs work » Needs review
StatusFileSize
new4.95 KB

Here's a patch that adds an environment variable and gives three options for handling PHP notices:

  1. Ignore. If the environment variable is missing or set to 0, there's no change to the test process and no messages or exceptions are created.
  2. Fail the step where the notice is generated. The PHP notice will be output to the screen and appended to a local file, a screenshot will be taken, and an exception will be thrown.
  3. Don't fail individual tests, but fail the suite. The PHP notice will be output to the screen and appended to a local file, a screenshot will be taken, but no exception will be thrown in the individual test. Instead, an exception will be thrown in the AfterSuite.

The patch also re-categorizes any "php" entries generated by the tests in the watchdog as "behat" entries, so they will persist across scenarios. This also means they'll still be in the dblog for review after the test suite is over, not cleared until somebody runs another test suite.

cboyden’s picture

Issue summary: View changes
StatusFileSize
new4.63 KB

After some more testing, I've updated the patch to change what the various options do.

  1. Ignore. If the environment variable is missing or set to 0, there's no change to the test process and no messages or exceptions are created.
  2. Print notices only. Any PHP notices will be output to the screen at the end of each scenario (not step) and appended to a local file. No exceptions will be thrown; if there are no other failures, the scenarios and suite will pass. There will be a message printed at the end of the suite saying that PHP notices were logged.
  3. Print notices and throw an exception in the AfterScenario. The PHP notice will be output to the screen and appended to a local file, and the scenario will fail. The complication with this is that in the current Panopoly setup, the whole suite will run again because there were failures, but --rerun will not detect them as failures.

  • dsnopek committed 32600e2 on 7.x-1.x authored by cboyden
    Issue #2361037 by cboyden: Detect PHP notices/errors
    
dsnopek’s picture

Status: Needs review » Fixed

This looks awesome!

Switching the 'php' watchdog messages to 'behat' is an interesting way to do it. I think you could have also simply set a member variable in flagPhpScenarioErrors() that you then checked in flagPhpSuiteErrors() to tell if there were any PHP notices, and simply cleared them at the end of each scenario. But this is fine. If you're running tests locally you can even still look in the recent log messages and see all of them, which could be useful.

I tested locally (with all possible values of the environment variable), as well as on Travis, both not setting the environment variable and setting it to 1 to print the messages. The latter way is how I'm going to merge this into Panopoly, so that we can start working on fixing all of our PHP notices.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.