Problem/Motivation

When running tests with filtering (e.g. by test suite, or by group), run-tests.sh Drupal own runner and PHPUnit CLI runner should ideally discover the same list of test classes to be executed. This is currently not happening, because:

  1. Some core tests classes do not have a name that ends with *Test. These tests are found by run-tests.sh but not by PHPUnit.
  2. Some core tests classes are placed in directories that PHPUnit 10 cannot reach via the current phpunit.xml.dist configuration. These are tests placed in core submodules. Unfortunately, usage of the globstar pattern like <directory>modules/**/tests/src/Unit</directory>
  3. was only introduced in PHPUnit 11, https://github.com/sebastianbergmann/php-file-iterator/pull/83. Until then, the only possibility is to explicitly indicate the path in phpunit.xml.dist.

Steps to reproduce

Proposed resolution

Fix the issues and add a test that continuosly compares the result of PHPUnit and TestDiscovery test discovery lists and fails if they differ.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#4 3477634-nr-bot.txt12.66 KBneeds-review-queue-bot

Issue fork drupal-3477634

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mondrake created an issue. See original summary.

mondrake’s picture

Status: Active » Needs review
needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new12.66 KB

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

mondrake’s picture

Status: Needs work » Needs review

rebased

mondrake’s picture

Rebased and added <directory>modules/**/tests/src/Build</directory> to build test suite.

smustgrave’s picture

How come PhpUnitCliTest is removed?

mondrake’s picture

#7 it's replaced by PhpUnitTestDiscoveryTest , which is a kernel test that does the same thing (call PHPUnit CLI to get the test list), but needs services to build Drupal's list to compare.

daffie’s picture

Status: Needs review » Needs work

When I run the Drupal\KernelTests\Core\Test\PhpUnitTestDiscoveryTest on my local machine it fails. THe code dump(array_diff(array_values($internalList), array_values($phpUnitList))); returns:

array:35 [
  1439 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestJsonAnonTest"
  1440 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestJsonBasicAuthTest"
  1441 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestJsonCookieTest"
  1442 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestXmlAnonTest"
  1443 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestXmlBasicAuthTest"
  1444 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestXmlCookieTest"
  1635 => "Drupal\Tests\demo_umami_content\Functional\DefaultContentFilesAccessTest"
  1636 => "Drupal\Tests\demo_umami_content\Functional\UninstallDefaultContentTest"
  1637 => "Drupal\Tests\drupal_system_listing_compatible_test\Kernel\SystemListingCrossProfileCompatibleTest"
  1660 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleJsonAnonTest"
  1661 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleJsonBasicAuthTest"
  1662 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleJsonCookieTest"
  1663 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleXmlAnonTest"
  1664 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleXmlBasicAuthTest"
  1665 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleXmlCookieTest"
  1666 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestComputedFieldNormalizerTest"
  1667 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestJsonAnonTest"
  1668 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestJsonBasicAuthTest"
  1669 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestJsonCookieTest"
  1670 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestJsonInternalPropertyNormalizerTest"
  1671 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelJsonAnonTest"
  1672 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelJsonBasicAuthTest"
  1673 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelJsonCookieTest"
  1674 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelXmlAnonTest"
  1675 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelXmlBasicAuthTest"
  1676 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelXmlCookieTest"
  1677 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestMapFieldJsonAnonTest"
  1678 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestTextItemNormalizerTest"
  1679 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestXmlAnonTest"
  1680 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestXmlBasicAuthTest"
  1681 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestXmlCookieTest"
  2282 => "Drupal\Tests\layout_builder_expose_all_field_blocks\Functional\GenericTest"
  2654 => "Drupal\Tests\navigation_top_bar\Functional\GenericTest"
  2816 => "Drupal\Tests\package_manager\Build\PackageInstallTest"
  2817 => "Drupal\Tests\package_manager\Build\PackageUpdateTest"
]

The code changes in the MR look good to me.

mondrake’s picture

@daffie have you checked your local phpunit.xml vs the one in the MR?

mondrake’s picture

Status: Needs work » Needs review

rebased, still green.

@daffie this test will work on a vanilla install; if you have custom/contrib modules, it could very well be that there are tests that PHPUnit cannot discover (see the IS point 2). In any case, could you rather share PHPUnit's output than the dump?

daffie’s picture

Status: Needs review » Reviewed & tested by the community

@daffie have you checked your local phpunit.xml vs the one in the MR?

Oeps, I did not.

Now that I did do that the test passes on my local machine.
All code changes look good to me.
For me it is RTBC.

@modrake: Thank you for your help.

quietone’s picture

mondrake’s picture

rebased for baseline conflicts

  • larowlan committed 69f22495 on 11.x
    Issue #3477634 by mondrake, daffie: Ensure run-tests.sh and PHPUnit CLI...
larowlan’s picture

Status: Reviewed & tested by the community » Fixed

Committed to 11.x, thanks for this - will be great to see these caught automatically from here on.

wim leers’s picture

This has bitten me many times before, so this is awesome to see! 👏🙏

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.