Problem/Motivation
From #3217904-95: [meta] Support PHPUnit 10 in Drupal 11.
PHPUnit 10 drops support for the 'test suites' concept. https://github.com/sebastianbergmann/phpunit/commit/36354a9cdc69ecabd20a...
original issue that drove the requirements for this discovery mechanism.
#2499239: Use test suite classes to discover different test types under phpunit, allow contrib harmony with run-tests
Proposed resolution
Deprecate own implementation of test suites.
Remaining tasks
User interface changes
API changes
Data model changes
Release notes snippet
| Comment | File | Size | Author |
|---|
Issue fork drupal-3402444
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3402444-deprecate-test-suites
changes, plain diff MR !5452
Comments
Comment #3
mondrakeThe deprecations only affect the test of the feature itself.
Comment #4
spokjeLeft a question in the comment, but I think this is ready for inspection by the core committers.
Comment #5
neclimdulI'm not sure I agree with the understanding of the issue.
1. run-tests is _one_ way to run tests against Drupal. The existence and documentation of the use of core/phpunit.xml.dist is the other. Run-tests is so clunky, awkward, and slow I haven't used it in 10 years or more at this point? It works for testbot but I can't figure out how anyone would use it regularly.
2. moving away from exposing our tests to phpunit blocks deeper integration into the phpunit ecosystem which I feel like is _not_ where we want to be heading. Not running tests in your IDE natively, not being able to look at paratest, etc.
So we don't need to deprecated it, we need to replace it.
Comment #6
neclimdulComment #7
mondrake#5. Let me try clarifying since it's not clear.
run-tests.sh. Test suites are not used by it.phpunit.xml.distonly by identifying directories to be included in a suite, https://docs.phpunit.de/en/10.4/organizing-tests.html#composing-a-test-s..., no longer by passing a file that implement classes extending from\PHPUnit\Framework\TestSuite, which contain astatic function suite()that explode the supported directories. This makes Drupal's implementation of the*TestSuiteclasses unproductive in PHPUnit 10. That's at least my understanding of https://github.com/sebastianbergmann/phpunit/commit/36354a9cdc69ecabd20a...So the replacement in PHPUnit 10, independently from run-tests.sh, would be to remove the cruft implementation classes and replace (if possible) with a direct indication of the directories to scan in
phpunit.xml.dist.Hence deprecation here.
Comment #8
neclimdulWe originally tried to do this in directory definitions but eventually had to give up because it was so dynamic and varied with the structures Drupal can exist with. PHPUnit is also pretty limited in how you can define these. I don't think that requirement hasn't really changed so I don't think we can deprecate without a replacement.
I don't think this is a problem though, if we're clever, we can probably move the logic into the events available in PHPUnit 9+ with probably opens up a lot better functionality and growth. We should even be able to do a deprecation in Drupal 10 without converting fully to PHPUnit 10. Just triggered if the static method is called directory and then pointing users to update their local configuration along the timelines of the PHPUnit 10 to what ever the event system looks like.
Comment #9
mondrakeTime will tell. Anyway we agree these classes will have to go away, right?
Comment #10
neclimdulSo yeah, I was hopeful about the event and extensions but after tracing up and down there's not much to be done. The classes need to go away, just not sure what the path forward for supporting our various testing logic. Mostly discovering tests in submodules and contrib.
Added the original that added discovery logic to summary.
Comment #11
mondrakeMR still applies, up for review again then
Comment #12
spokjeComment #13
longwaveThe deprecations need a change record to point to.
Comment #14
mondrakeDraft CR @ https://www.drupal.org/node/3405829
Comment #15
mondrakeAdjusted references to CR and rebased.
Comment #16
neclimdulI'm not sure I was being clear since this is back to RTBC but my concerns aren't addressed.
Because of the way this is structured and the way the message is worded this looks like we're deprecating phpunit entirely.
The problem is, there's no replacement for the class, but we should have a replacement in PHPUnit 10 for running PHPUnit tests.
I was asking that we figure out what that replacement is, then provide that explanation in the message and CR instead of just saying there's no replacement.
Comment #17
mondrakeNW then.
Comment #18
neclimdulMaybe to limit the scope, keep this clean, and not build a circular dependency between this and the larger PHPUnit upgrade we can do
s/There is no replacement/There is no replacement and test discovery will be handled differently in PHPUnit 10./and then the CR has some block about following the PHPUnit 10 upgrade issue. At least if that's ok with committers.We can then clarify everything when finalize the upgrade path.
Comment #19
catchI think #18 is OK, the change record can point out this doesn't affect running individual tests locally (or run-tests.sh on gitlab) etc.
Do we need a new follow-up issue to figure out what the replacement is or is one already open? There's quite a few notes in the issue summary of #3217904: [meta] Support PHPUnit 10 in Drupal 11 which could be copied over along with findings from here.
Comment #20
mondrakeChanged deprecation message.
Comment #21
smustgrave commentedSeems to have phpcs failures.
Comment #22
mondrakeSpell-checking… I cannot explain the job failure though, seems related to git branches, not to real spelling errors
Comment #23
longwaveRebase the branch against 11.x; the spellchecking errors have been solved in the core repo but GitLab CI runs the scripts from when the MR was branched.
Comment #24
spokjeJust did that ^
:)
Comment #25
longwaveComment #26
spokjeAll green now (as @longwave already predicted above apparently ;)
Comment #27
mondrakeThanks @Spokje! Mind sharing what commands did you use to get that?
Comment #28
spokjeThe rebase button/link has disappeared in MRs, but just adding
/rebaseas a comment on the MR (so in this case: https://git.drupalcode.org/project/drupal/-/merge_requests/5452) will auto-rebase if there aren't any conflicts that GitLab can't solve without manual intervention.If the latter is the case, the command will do nothing.
Comment #29
mondrakeTIL^^
Thanks!!
Comment #30
smustgrave commentedI also was not aware of that thanks!
I believe feedback has been addressed though
Comment #31
spokjeNote on the TIL: You must have push-access on the MR for the magic to work.
Comment #32
longwaveCommitted dccd322 and pushed to 11.x. Thanks!
Also published the change record. See you all in the next PHPUnit 10 issue!
Comment #36
catchThis broke the performance test job on gitlab: https://git.drupalcode.org/project/drupal/-/jobs/788445 Not sure exactly why yet but reverted for now.
Comment #37
mondrakeI think that’s because those tests are run directly via Phpunit CLI
—groupand not via run-tests.sh.That triggers PHPUnit’s own test discovery that loads all the classes and therefore triggers the deprecation errors. run-tests uses its own test discovery and then runs test on a file-by-file basis, so that doesn’t occur.
Not sure what is the best way forward. Maybe just avoid the
@trigger_errorof the deprecation of the classes so we don’t get this issue with autoloading.Comment #39
mondrakeThis could do, I think.
We remove @trigger_error on classes, that fires on class autoloading, leaving the @deprecate annotations. That's enough to have PHPStan detect usage of deprecated classes, in case - that's demonstrated by the fact that we need to add to PHPStan baseline the extension of TestSuiteBase by its child classes.
Comment #40
quietone commentedI've unpublished the CR
Comment #41
smustgrave commentedSo what was the solution for the problem seen in #36? Or still being determined?
Comment #42
mondrake#37 is the explanation, and #39 is the solution. Is that not clear enough?
Comment #43
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #44
mondrakeCome on, bot…
Comment #45
catchLooks good to me, will commit later if no comments since this already landed once.
Comment #46
spokjeComment #47
catchNeeds a rebase.
Comment #48
mondrakerebased, no clashes
Comment #51
catchCommitted/pushed to 11.x and 10.3.x, thanks!
Comment #53
mondrakePublished the CR. As to ‘what now, then?’, the MR in #3417066: Upgrade PHPUnit to 10, drop Symfony PHPUnit-bridge dependency has now a proposal to include the files to be included in the suites, that covers core lib tests, core module tests, and tests for modules that are placed in the /module subdirectory. I think the CR will have to be updated once that eill be confirmed to be the final approach for D11/PHPUnit 10.