Problem/Motivation
When running tests with filtering (e.g. by test suite, or by group), run-tests.sh Drupal own runner and PHPUnit CLI runner should ideally discover the same list of test classes to be executed. This is currently not happening, because:
- Some core tests classes do not have a name that ends with
*Test. These tests are found by run-tests.sh but not by PHPUnit. - Some core tests classes are placed in directories that PHPUnit 10 cannot reach via the current
phpunit.xml.distconfiguration. These are tests placed in core submodules. Unfortunately, usage of the globstar pattern like<directory>modules/**/tests/src/Unit</directory>
was only introduced in PHPUnit 11, https://github.com/sebastianbergmann/php-file-iterator/pull/83. Until then, the only possibility is to explicitly indicate the path in phpunit.xml.dist.
Steps to reproduce
Proposed resolution
Fix the issues and add a test that continuosly compares the result of PHPUnit and TestDiscovery test discovery lists and fails if they differ.
Remaining tasks
User interface changes
Introduced terminology
API changes
Data model changes
Release notes snippet
| Comment | File | Size | Author |
|---|
Issue fork drupal-3477634
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3477634-ensure-run-tests.sh-and
changes, plain diff MR !9684
Comments
Comment #3
mondrakeComment #4
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #5
mondrakerebased
Comment #6
mondrakeRebased and added
<directory>modules/**/tests/src/Build</directory>to build test suite.Comment #7
smustgrave commentedHow come PhpUnitCliTest is removed?
Comment #8
mondrake#7 it's replaced by
PhpUnitTestDiscoveryTest, which is a kernel test that does the same thing (call PHPUnit CLI to get the test list), but needs services to build Drupal's list to compare.Comment #9
daffie commentedWhen I run the Drupal\KernelTests\Core\Test\PhpUnitTestDiscoveryTest on my local machine it fails. THe code
dump(array_diff(array_values($internalList), array_values($phpUnitList)));returns:The code changes in the MR look good to me.
Comment #10
mondrake@daffie have you checked your local phpunit.xml vs the one in the MR?
Comment #11
mondrakerebased, still green.
@daffie this test will work on a vanilla install; if you have custom/contrib modules, it could very well be that there are tests that PHPUnit cannot discover (see the IS point 2). In any case, could you rather share PHPUnit's output than the dump?
Comment #12
daffie commentedOeps, I did not.
Now that I did do that the test passes on my local machine.
All code changes look good to me.
For me it is RTBC.
@modrake: Thank you for your help.
Comment #13
quietone commentedComment #14
mondrakerebased for baseline conflicts
Comment #16
larowlanCommitted to 11.x, thanks for this - will be great to see these caught automatically from here on.
Comment #18
wim leersThis has bitten me many times before, so this is awesome to see! 👏🙏