Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.If I run FunctionalJavascript tests when ChromeDriver is not running, the tests always pass. Instead they should fail with the message that ChromeDriver is not running.
Original report
The following test ought to fail in three separate ways, but it passes when I run this test.
namespace Drupal\Tests\my_module\FunctionalJavascript;
use Drupal\FunctionalJavascriptTests\WebDriverTestBase;
/**
* Basic test.
*
* @group MyTest
*/
class MyTestBasicTest extends WebDriverTestBase {
protected $defaultTheme = 'stark';
/**
* Basic test.
*/
public function testPage() {
$this->drupalGet('');
$this->assertSession()->pageTextContains('Text that is not on the page.');
non_existant_function();
exit;
}
}
This is the test output:
Status Group Filename Line Function
--------------------------------------------------------------------------------
Pass Other MyTestBasicTest.p 19 Drupal\Tests\my_module\FunctionalJa
It finds the test method on line 19, but it doesn't run it. Does anyone know what the problem is?
| Comment | File | Size | Author |
|---|---|---|---|
| #7 | 3187577-7.patch | 1.17 KB | longwave |
Comments
Comment #2
longwaveWhat command line are you using to run the test?
Comment #3
liam morlandphp run-tests.sh --verbose --color --non-html --sqlite /tmp/drupal/test.sqlite --module my_module --url <url>Comment #4
liam morlandI have observed this problem in core tests as well. For example, in 9.2.x, I edited
core/tests/Drupal/FunctionalJavascriptTests/TableDrag/TableDragTest.phpby adding a call to a non-existent function to one of the test methods. That ought to make it fail. The following command still resulted in the tests passing:php core/scripts/run-tests.sh --verbose --color --non-html --sqlite /tmp/drupal/test.sqlite --url <url> --class "Drupal\FunctionalJavascriptTests\TableDrag\TableDragTest"If I edit
TableDragTest.phpso that it is invalid PHP, such as by removing a line-ending semi-colon, I get a PHP parse error.Comment #5
liam morlandThe immediate problem is that ChromeDriver is not running. If I run
/usr/bin/chromedriver --port=4444prior to running therun-tests.shcommand above, then the tests fail as expected.Comment #6
liam morlandIf it can't connect to ChromeDriver, PHPUnit will skip the test (this can be seen by running
vendor/bin/phpunitdirectly).run-tests.shtreats this as a pass.Comment #7
longwaveSo it feels like we should hard fail instead of skip if an error occurs during Mink setup.
Running this locally without a Chromedriver I get:
Comment #8
liam morlandThe patch works.
Perhaps it would be better if the skip status bubbled-up so that
run-tests.shshows the test as skipped rather than failed. The exit status ofrun-tests.shshould be non-zero if any tests do not pass, whether skip or fail.Comment #9
longwaveTest skips should not be considered as fails by the runner. We otherwise use skips to denote that a test is not compatible with a particular environment, e.g. a test for MySQL being run on a different database driver, or a deprecation test for PHPUnit that is being run on a version where the code is already removed. This case is a real failure because we couldn't even run the test at all, it is not due to a compatibility issue.
Comment #10
liam morlandAh, that makes sense.
Comment #11
alexpottHonestly I don't think anyone outside of DrupalCI should be using run-tests.sh - if you ran with phpunit the problem would be obvious. Having a running chromedriver is a requirement of the test. The test itself has not failed. Reporting a fail is incorrect. See https://phpunit.de/manual/6.5/en/incomplete-and-skipped-tests.html - for me this is the same as not having PHP extension that is required or a database server.
I think this issue is a duplicate of #2905007: Allow run-tests.sh to report skipped/incomplete PHPUnit tests.
Comment #12
liam morlandAre you saying that when we run our custom tests for our web site we should use phpunit directly instead of using run-tests.sh?
Comment #13
alexpott@Liam Morland that's what I do. One less layer.
Comment #16
liam morlandI have updated the documentation to inform people that they should use phpunit directly instead of using run-tests.sh.