Simpletest fails to run PHPUnit on Windows

The same is true on *nix platforms when you download a tar.gz, because there vendor/bin/phpunit is no longer a symlink and then the relative path to find autoload.php fails

I found this issue when I was about to file a new one for what I believe to be the same issue. Please correct me if I'm wrong.

On Linux, running ./vendor/bin/phpunit only works from a Git checkout and not from within a codebase extracted from a tarball. To temporarily fix the issue, you have to do the following from within the vendor/bin directory:

$ wget http://getcomposer.org/composer.phar && touch composer.json

Add the below lines to your new composer.json file:

{
    "require": {
        "phpunit/phpunit": "4.1.*",
        "phpunit/phpunit-mock-objects": "dev-master#e60bb929c50ae4237aaf680a4f6773f4ee17f0a2"
    }
}

Then run:

$ php composer.phar install

Obviously, now that I know this is related to a wrong symlink it all makes more sense to me. I did try to apply #4 on a fresh tarball (from July, 9th) and it unfortunately didn't help.

$ curl https://www.drupal.org/files/issues/test.phpunit-windows.4.patch | git apply -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1600  100  1600    0     0   1792      0 --:--:-- --:--:-- --:--:--  1791
Checking patch core/modules/simpletest/simpletest.module...
Hunk #1 succeeded at 287 (offset 46 lines).
Hunk #2 succeeded at 300 (offset 46 lines).
Applied patch core/modules/simpletest/simpletest.module cleanly.

Did I miss anything?

I didn't experience any issue with run-tests.sh and was under the impression that this issue was addressing problems with vendor/bin/phpunit only?

What exactly is not working for me is running PHPUnit tests (as described here) from an extracted tarball while it works flawlessly from within a 8.x git clone.

$ for i in tarball head ; do echo ========== ; cd /var/www/html/$i/core && ./vendor/bin/phpunit ; done
==========
You need to set up the project dependencies using the following commands:
wget http://getcomposer.org/composer.phar
php composer.phar install
==========
PHPUnit 4.1.3 by Sebastian Bergmann.

Configuration read from /var/www/html/head/core/phpunit.xml.dist

.............................................(snipped)

In the above testing scenario, I don't have the patch applied and it fails immediately from within a tarball and succeeds with a git checkout.

Which is sad, because we have been documenting and communicating the other for a long time now and it only broke with PHPUnit 4 AFAIK.

The only reason it breaks in the tarball is because the relative path inside is is no longer correct. It's already checking in two locations, would be easy to add a third.

IMHO, this is a bug in PHPUnit and should be fixed there, either by not using a symlink for this in a first place and/or fixing the path where it looks for autoload.php.

And run-tests.sh is a *very* crappy DX for running unit tests, it's slow, has no filter possibilities or any of the other useful arguments and the output is much worse, so we really shouldn't suggest to use that for unit tests.

Thanks for the clarification, now I understand the confusion. I don't want to hold up this patch either and we can update the documentation afterwards.

It's only a pity (as in less elegant) that ./core/vendor/bin/phpunit be replaced with ./core/vendor/phpunit/phpunit/phpunit but we can live with it :-)

core/vendor/phpunit/phpunit/phpunit will always be a canonical location for the PHPUnit binary. bin/ just saves some typing, and lets you have a place for PATH to use.

The work-around is to delete core/vendor/ and then do composer install whenever you download Drupal and want to run tests.

I'd say this issue demands some attention, but I don't have Windows and can't review the patch.

Maybe this will help: #2036705: Drupal 8 doesn't work well on WAMP stack.

Or maybe this will help: #2389287: Missing PhpExecutableFinder.

The patch no longer applies.

Every unit test I'm running ends with "No test results to display." There is no change with this patch applied. Does this have any installation requirements in XAMPP?

@hass
The only sane way to run the phpunit tests are using the command line.

For the testbot to work.

Patch no longer applies. Bummer. I would attempt to reroll but I'm really not sure what's going on here.

Rerolled.

This is great, thanks! I'll definitely test, just need to find a moment.

@othermachines, If you could figure out why SimpleTestBrowserTest is failing, that would be wonderful! I haven't had time to look into that failure yet and Windows-only (ok, not technically correct in this case) bugs don't get a lot of attention from the rest of the crowd.

Thanks!

I've swapped the params in join() which should (hopefully!) take care of it.

Well, there. I didn't know join() could accept args in any order. But I'm pretty certain we don't need an implode *and* a join. Do we need to change this line? SimpletestPhpunitRunCommandTest passes if we leave it as is. Err, that's good right?

As for SimpleTestBrowserTest the rest of it, beats me. Right now all I'm getting is an internal server error and a mess to clean up afterwards. Argh.

I'm pretty sure $ret should contain an integer, not the last line of the output, which is what you get with $ret = exec(join(" ", $command), $output);.

I can't test this locally at the moment but I'm curious to know if it will pass.

But I'm pretty certain we don't need an implode *and* a join. Do we need to change this line?

Yes, I believe I added that in #29 in what can only be described as a total reroll failure! Clearly working too late... :(

Thanks for continuing to work on this -- I'll be able to review your last patch in the morning.

Well, terrific. With latest patch I can now run PHPUnit tests via UI on Windows. To be on the safe side I ran the same tests from the command line and they all sync up.

Sorry, haven't had the time to test the latest patch... Regardless, the issue summary needs and update and beta eval, which would help folks who are starting to look at this issue.

Potentially needs docs update as per #9 with much of that @sun mentions in #11, though I'm not sure what @sun says in #11 is what we want to use as our default way to run phpunit. As clunky as it is cd core; ./vendor/phpunit/phpunit/phpunit ... is the only guaranteed way. Anyhow, though I through that out there in case someone disagrees.

There are other serveral thing broken with simpletest. Adding related and/or possible duplicate issue.

#2605284: Testing framework does not work with contributed database drivers

The symlink wouldn't clone in an OSX 10.10 environment when I was working with vdd Vagrant setup virtualizing Linux. I filed a bug here. It's easy enough to fix by making a link manually but enough a problem to be worth filing a bug. #2659650: Cloning Drupal 8.0.x head: unable to create symlink vendor/bin/phpunit (Protocol error)

Given that we use composer now to deal with the dependencies I guess this is no longer an actual issue.

(removing beta eval tag since we're no longer in beta...)

Sorry, @dawehner, I wish that were the case but running PHPUnit tests via the Simpletest UI still fails on Windows because it calculates the wrong path the PHPUnit executable:

Without the patch:

With the patch:

Note that on Windows (and maybe other platforms -- haven't had a chance to investigate and update docs) one needs to set their PHP_PATH environment variable to the PHP executable they want to use. It could be considered in-scope for this issue to have a better error message if this is not set. (Currently fails with a blank error.)

Re-uploading #40 for the new testbot.

Note that on Windows (and maybe other platforms -- haven't had a chance to investigate and update docs) one needs to set their PHP_PATH environment variable to the PHP executable they want to use.

Are you sure it's the PHP binary and not a problem finding the PHPUnit executable?

If so, it's similar to this: #2476139-17: Fix UI test fail in SimpleTestBrowserTest

Are you sure it's the PHP binary and not a problem finding the PHPUnit executable?

Yeah, pretty sure. When I was stepping through the code it was the PhpExecutableFinder that was failing to return an valid php.exe. In other words,

+++ b/core/modules/simpletest/simpletest.module
@@ -310,17 +310,18 @@ function simpletest_phpunit_command() {
     $php = $php_executable_finder->find();

would return null.

@mikeker
What you are experiancing is part of #2748883: Use the PHP_BINARY constant in addition to --php in run-tests.sh

With the latest patch, tests properly run from both UI and run-tests.sh

+++ b/core/modules/simpletest/simpletest.module
@@ -310,17 +310,18 @@ function simpletest_phpunit_command() {
+  $phpunit = $vendor_dir . '/phpunit/phpunit/phpunit';
   if (substr(PHP_OS, 0, 3) == 'WIN') {
...
   else {
-    $phpunit_bin = $vendor_dir . '/phpunit/phpunit/phpunit';
+    $command = $phpunit;
   }
-  return $phpunit_bin;
+  return $command;

If we call this variable command then we don't need the else below. So the result code is simpler. Less logic to stare at and wonder what is going on.

Like so.

Tested on Cli & UI.

Why are we running the PHP executable through escapeshelllargs?

+ $command = escapeshellarg($php) . ' -f ' . escapeshellarg($command) . ' --';

This breaks the fact that the PHP executable ($php) itself can have arguments. It's not just the executable.

I.e:

$php = "c:\tools\php\php.exe";

or

$php = "c:\tools\php\php.exe -c \"c:\myscutomphp.ini\""

Current escapeshellargs on $php will break second usage.

This should do it. BTW, @droplet, great find about --printer argument breaking autolading.

Tests not working on an OS that has > 80% desktop market share should be Major.

Just a tidy clean up of how we're getting the classname.

Lost track of this issue, nice to see it moving forward. FWIW tests pass for me from UI and CLI with #63 applied (8.4.x branch). Cheers and thx -

Committed and pushed 41eda3b to 8.4.x and ec943f0 to 8.3.x. Thanks!

This is also worth mentioning in the testing improvements in the release notes/changelog, I think.