[meta] DrupalCI/SimpletestUI/run-tests.sh/phpunit not in harmony

Problem/Motivation

This issue summary is based on a comment from here: https://www.drupal.org/node/2499239#comment-10520662 #2499239: Use test suite classes to discover different test types under phpunit, allow contrib harmony with run-tests where it seems that I alone am the only person who cares about being able to run unit tests from the command line for contrib with submodules.

Here's an issue that added a --directory option to the D8 run-tests.sh script: #2551981: Add --directory option to run-tests.sh test discovery

This means that when it comes time to discover tests, run-tests.sh and phpunit have different behaviors, because run-tests.sh ignores phpunit.xml.dist.

Similarly, SimpletTest in the UI finds some tests that phpunit doesn't, and phpunit finds some tests that SimpleTest in the UI doesn't.

I would wager that there is a discrepancy between the tests run by run-tests.sh and SimpleTest UI as well, but I don't have that information.

So that leaves us with the testbot finding files that I can't run locally, leading to a false positive locally.

From the issue for the patch above (#2551981: Add --directory option to run-tests.sh test discovery) @jthorson says:

However, instead of building this scanning capability into the new test runner, the maintainers assert that test discovery should be the responsibility of the test framework itself, not the individual runner ... i.e. the scanning for files should be the responsibility of run-tests.sh, not PIFR or DrupalCI.

Absolutely it should be the responsibility of the test framework, not the runner. But it shouldn't be up to run-tests.sh to ignore PHPUnit configuration.

So, just to tally, we have:

1) SimpleTest in the UI which *does* discover and run the submodule PHPUnit tests, according to.... policy?

2) run-tests.sh which uses phpunit.xml.dist for test configuration such as strict rules (I think) but which scans for tests everywhere, depending on calling mode.

Which leaves us with:

3) PHPUnit at the command line, which honors whitelist/blacklist from phpunit.xml.dist, meaning it *can't* run tests that the others *can*, for... reasons?

Sadly, the only useful information you'll get about PHPUnit tests comes from running phpunit at the console, which means you're SOL if you want to find out why a test failed. Here's what run-tests.sh tells you when you pass --verbose:

Detailed test results
---------------------


---- Drupal\Tests\phpunit_example\Unit\AddClassTest ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      AddClassTest.php    55 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A
    
Pass      Other      AddClassTest.php     0 Drupal\Tests\phpunit_example\Unit\A

$

Proposed resolution

Desired behavior:

$ ./vendor/bin/phpunit -c core --group my_contrib_submodule

Since @jthorson is right, and it should be up to D8 to determine what tests to run, we should design that.

We should have a way to determine what tests to run, both for core and contrib.

We might have a phpunit.xml.core and a phpunit.xml.contrib if needed.

Add your idea here.....

Remaining tasks

User interface changes

API changes

Data model changes