Test-runner ignores @group annotations for PHPUnit-based tests in modules.

This is a duplicate of #2427191: Test-runner ignores @group annotations for PHPUnit-based tests in modules.

I guess you meant #2301481: Mark test groups as belonging to modules in UI, but I'm not so sure, depends on what we end up doing? This is about having a huge PHPUnit test group, instead of splitting this up. The other one is about action vs. Action test groups and how they are sorted...

Reopening per #2.

Just a note on the beta eval -- there is some disruption, because it would require changes in many patches and subsystems for core to be consistent. (See https://www.drupal.org/core/beta-changes#disruption). Probably not a big deal, but we should be explicit. :)

Thanks for giving it some thought already though!

Also not really sure it's a bug actually. It's a design behavior. Whether it's a good one is another question. ;) But also improvements to testing are less frozen than other things so worth exploring.

Ah... I must have spent 2 hours trying to determine why my unit tests were not grouping under my module name instead of get dumped under PHPUnit. Right now it is not clear at all why that is, when all other tests look nicely grouped.

Would it be unreasonable to have an integration test list and a unit test list? I can foresee that PHPUnit group being enormous when you start adding in contrib.

Ideally we'd solve #2297541: [policy, no patch] Secondary/additional test @group names

@kevinquillen tbh PHPUnit tests are best run using the PHPUnit runner rather than simpletest.

Also not really sure it's a bug actually.

Bug.

Definitely bug.

Try this: Find a module with PHPUnit tests in it.

Run run-tests.sh with that module name as the group.

Note that the unit tests were not run even if they were annotated as @group modulename.

Then use the --module option with the module name.

Note that the unit tests *are* run.

For instance:

$ php ./core/scripts/run-tests.sh action

Drupal test run
---------------

Tests to be run:
  - Drupal\action\Tests\ActionUninstallTest
  - Drupal\action\Tests\BulkFormTest
  - Drupal\action\Tests\ConfigurationTest
  - Drupal\action\Tests\Migrate\d7\MigrateActionsTest

Test run started:
  Monday, February 1, 2016 - 06:13

Test summary
------------

Note that it misses Drupal\Tests\action\Unit\Menu\ActionLocalTasksTest and Drupal\Tests\action\Unit\Plugin\migrate\source\ActionTest, both of which are annotated with @group action.

You can run either of those tests individually using --class, and you can also run all the tests including those ones with --module action:

$ php ./core/scripts/run-tests.sh --module action

Drupal test run
---------------

Tests to be run:
  - Drupal\action\Tests\ActionUninstallTest
  - Drupal\action\Tests\BulkFormTest
  - Drupal\action\Tests\ConfigurationTest
  - Drupal\action\Tests\Migrate\d7\MigrateActionsTest
  - Drupal\action\Tests\Migrate\d6\MigrateActionConfigsTest
  - Drupal\action\Tests\Migrate\d6\MigrateActionsTest
  - Drupal\Tests\action\Unit\Menu\ActionLocalTasksTest
  - Drupal\Tests\action\Unit\Plugin\migrate\source\ActionTest

Test run started:
  Monday, February 1, 2016 - 06:18

Test summary
------------

Note that this finds *all* tests in the module, regardless of annotation, which is reasonable. However, this also proves that the unit tests are discovered but filtered out despite their group annotation in the previous example.

This is inconsistent behavior. Bug.

The problem is that TestDiscovery OBLITERATES the group annotation while building the list and arbitrarily assigns it to @group PHPUnit so that it looks pretty in the SimpleTest UI. run-tests.sh uses this same discovery path for groups (but obviously not for modules). Which is inconsistent and thus buggy.

The way to solve this is to fix the bug that lives right here: http://cgit.drupalcode.org/drupal/tree/core/modules/simpletest/src/TestD...

The reason that bug is there is so that you can group PHPUnit-based tests in one place in the SimpleTest UI and exclude them from running in SimpleTest. That was to avoid early problems running PHPUnit-based tests alongside SimpleTest ones.

Fixing this bug, combined with #2608532: Simpletest UI munges PHPUnit-based test output would mean everyone could run phpunit based tests happily and usefully from run-tests.sh and also from the SimpleTest UI.

Updating issue summary and title to reflect the actual problem. Step-by-step instructions to solve this problem are included. And what the heck, here's a patch.

You'll see that the patch isn't very big at all, is it?

It still flakes out in the SimpleTest UI for running the action group. It only runs the 4 SimpleTest-based tests, and misses the 2 phpunit tests despite their being checked to run in the form. This is a consequence of an error in how the UI runs unit tests. But that's outside the scope of this issue and inside the scope of: #2608532: Simpletest UI munges PHPUnit-based test output

It does fix the issue with run-tests.sh noted above.

Branch and patch both ran 14,475 tests, so we're not dropping or adding any.

Fixed by #2659100: Allow run-tests.sh to run just the javascript Functional tests