It could be useful to detect whether there are any PHP errors or notices being generated during Behat tests. A demonstration of the idea is here: https://www.godel.com.au/blog/use-behat-track-down-php-notices-they-take....
The suite could output a list of PHP notices at the end of the results, just before the summary of passed/failed scenarios. You could also do this after every scenario, but that might be too much overhead.
Patch to come.
| Comment | File | Size | Author |
|---|---|---|---|
| #6 | panopoly_test-php-notices-2361037-6.patch | 4.63 KB | cboyden |
Comments
Comment #1
cboyden commentedPatch is attached. This code truncates the watchdog table at the beginning of the suite and prints out all PHP notices etc. at the end.
Comment #2
cboyden commentedThere are a few options if you want PHP notices to have some effect on the exit status of automated tests.
On Travis, you could add a step after each suite that would check the watchdog. If PHP notices existed, you could set the exit status to error. Problems with this approach:
You could also adapt the above patch to write a file with the PHP errors for each suite (or scenario, that'd be more useful). Travis could check for the existence of files at the end of the run_tests step and set the exit status to error if they existed. Problems with this approach:
Comment #3
dsnopekHere's what I'd personally think would be awesome:
This way, everyone always gets to at least see the notices, and we can still make it an error in Travis.
What do you think?
Comment #4
lsolesen commentedComment #5
cboyden commentedHere's a patch that adds an environment variable and gives three options for handling PHP notices:
The patch also re-categorizes any "php" entries generated by the tests in the watchdog as "behat" entries, so they will persist across scenarios. This also means they'll still be in the dblog for review after the test suite is over, not cleared until somebody runs another test suite.
Comment #6
cboyden commentedAfter some more testing, I've updated the patch to change what the various options do.
Comment #8
dsnopekThis looks awesome!
Switching the 'php' watchdog messages to 'behat' is an interesting way to do it. I think you could have also simply set a member variable in
flagPhpScenarioErrors()that you then checked inflagPhpSuiteErrors()to tell if there were any PHP notices, and simply cleared them at the end of each scenario. But this is fine. If you're running tests locally you can even still look in the recent log messages and see all of them, which could be useful.I tested locally (with all possible values of the environment variable), as well as on Travis, both not setting the environment variable and setting it to 1 to print the messages. The latter way is how I'm going to merge this into Panopoly, so that we can start working on fixing all of our PHP notices.