With code coverage on, PHPUnit records every line a test runs. That can make code look tested when it isn't. For example, Url::toString() calls UrlGenerator to build the URL, so a test of Url also runs UrlGenerator's code. Those lines would show as covered, even though the test only checks Url. If UrlGenerator had no tests of its own, the coverage report would still make it look tested, and nobody would know its tests are missing.

To prevent that, a test can specify what code it actually tests with an attribute on the test class: #[CoversClass(Url::class)] for a class, #[CoversTrait] for a trait, #[CoversMethod] for one method and #[CoversFunction] for a function.

That specified code basically becomes the test's coverage target, and PHPUnit only counts lines in the target as covered by that test. UrlGenerator then has to get its coverage from its own tests.

PHPUnit measures the files listed in the <source> section of core/phpunit.xml.dist. Since PHPUnit 12, it also checks that every target is in that list.

For example, WebAssertTest has #[CoversClass(WebAssert::class)], but WebAssert lives in core/tests, which isn't in <source>. When a target is outside the list, PHPUnit throws away the test's coverage and reports a warning. Also, core sets failOnPhpunitWarning="true", and the run fails.

I've been working on making code coverage work and filed two other issues: #3630852: Tests whose coverage target is outside phpunit.xml.dist's source lose their coverage under PHPUnit 12 adds .module and .inc files and the composer/ directory to <source>, which fixes 13 tests. #3630772: PHPUnit 12 deprecated using #[CoversClass] to target traits fixes 25 tests that use #[CoversClass] for a trait, where PHPUnit 12 requires #[CoversTrait]. That leaves about 35 tests with targets that <source> leaves out on purpose:

  • 24 cover the test framework in core/tests, like KernelTestBase or WebAssert.
  • 8 cover a class in a test module.
  • 3 cover a class in core/lib/Drupal/Component, which core's phpunit.xml.dist excludes because the Component tests have their own.

So here is what we have to do:

  1. Run all of core's tests with coverage on (takes 8+ hours on my old laptop).
  2. Write down every test class with an "is not a valid target for code coverage" warning from PHPUnit.
  3. In each of those test classes, delete each Covers attribute that points to a class or method from the warnings. In WebAssertTest, for example, that's #[CoversClass(WebAssert::class)].
  4. If a test class has no Covers attribute left, add #[CoversNothing] to it, with use PHPUnit\Framework\Attributes\CoversNothing;, so the code it runs doesn't show up as covered.

(As an aside, Drupal Core already marks four tests #[CoversNothing]. The oldest is TestSiteApplicationTest, from 2018 (#2926633: Provide a script to install a Drupal testsite). The other three are Composer build tests, whose base class got it in 2022 (#3272110: Drupal 9 and 10's Drupal\Component composer.json files are totally out of date). None of those issues discussed adding #[CoversNothing] though.)

Issue fork drupal-3631031

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

dries created an issue. See original summary.

dries’s picture

I opened an MR that follows the steps in the issue summary. 32 tests get #[CoversNothing]. UrlTest, ConsoleTest and MigrateSqlSourceCountCacheTest still have another target, so they just lost the rejected ones.

However, two migrate tests needed an additional change. MigrateSourceTestBase::getPluginClass() decides which source plugin to test by reading the test's #[CoversClass]. In HighWaterNotJoinableTest and MigrateSqlSourceCountCacheTest, that attribute points to a plugin from a test module. To make PHPUnit happy, we have to remove the #[CoversClass], but with that removed, getPluginClass() falls back to code for the old @covers annotations, which calls a method PHPUnit 12 removed, and the tests errored. So the MR overrides getPluginClass() to return the plugin class directly.

FYI, the method is MetadataCollection::isCovers() which was removed in PHPUnit 12.0 because PHPUnit 12 dropped annotation support. My MR doesn't touch MigrateSourceTestBase, so the isCovers() call is still there. So I think we need a separate issue that deprecates finding the plugin from #[CoversClass] and gets rid of the MetadataCollection::isCovers() call? That would probably need a deprecation, because contributed modules can extend this base class for their own source plugin tests? In the mean time, my MR works around it, and is a step up in my opinion.

Anyway, with code coverage enabled, my run went from 254 PHPUnit warnings to none. Together with #3630852: Tests whose coverage target is outside phpunit.xml.dist's source lose their coverage under PHPUnit 12 and #3630772: PHPUnit 12 deprecated using #[CoversClass] to target traits, no test has a target outside <source> anymore, apart from the Component tests, which use their own phpunit.xml.dist.

AssertContentTraitTest and XdebugRequestTraitTest are also changed by #3630772: PHPUnit 12 deprecated using #[CoversClass] to target traits, so whichever MR lands second needs a rebase.

AI disclosure: I studied up on how PHPUnit's code coverage works and decided on the approach. Claude Code found and updated the 35 tests. I reviewed the diff before pushing it. I'm only 90% sure on the migrate tests changes though.

dries’s picture