Suppress deprecation warnings during test discovery to avoid failures due to test classes

Our test discovery is pretty special - making tests autoloadable is very very funky and something other projects don't do. I think we should do #2296635: Evaluate performance impact of limiting test discovery to *Test.php filename suffix instead of this issue.

I think this is hitting also #3127141: Use PHPUnit 9 for PHP 7.4+, while keeping support for PHPUnit 8.4 in PHP 7.3: during test discovery, data provider methods are executed to determine the tests to run, and if a deprecated method is called there, the deprecation does not get silenced.

Bumping to critical since this is along the way to PHP8 and PHPUnit9

So I couldn't work out why test discovery was causing the problem. And it's not. See #3172540: Don't skip LegacyMimeTypeGuesser deprecation anymore - the problem is being caused by

      # Re-run Composer plugin tests after installing Composer 2
      container_command.composer-upgrade:
        commands:
          - "sudo composer self-update --snapshot"
          - "./vendor/bin/phpunit -c core --group VendorHardening,ProjectMessage,Scaffold"
        halt-on-fail: true

in drupalci.yml - it is only the Composer 2 tests that are causing this problem. The deprecation is being triggered by Symfony's debug classloader because the test is being loaded to use reflection to discover the group. I think we should do something like the latest solution on #3172540: Don't skip LegacyMimeTypeGuesser deprecation anymore

Somewhat amusingly Drupal's test discovery that run-tests.sh uses doesn't cause this because we use static reflection (ie. we do not properly load the test). I know that once upon a time Sun tried to add static reflection to PHPUnit so that it did the same thing.

#4 @alexpott that is the visible test fail, yes - but if you look at the console log https://dispatcher.drupalci.org/job/drupal_patches/60872/testReport/ you see there's more to it, just those failures are not reported in the d.o. summary page. And all the failures are related to using ::prophesize in data provider methods.

@mondrake but those deprecations are real. We are using the prophesize method - imo we need to skip the deprecation until we're ready to upgrade all the calls. That's why we have that functionality. The deprecation in #3172540: Don't skip LegacyMimeTypeGuesser deprecation anymore is not real - it is only caused by phpunit's test discovery and our Compsoer 2 tests - it does not occur in the regular test run.

#7 ah sorry I was talking about #3127141-78: Use PHPUnit 9 for PHP 7.4+, while keeping support for PHPUnit 8.4 in PHP 7.3 and you about #3172540: Don't skip LegacyMimeTypeGuesser deprecation anymore.

but those deprecations are real

Yes, and given the patch they should be silenced. But they're not. The silencing works if the call to ::prophesize is in the test method, but it doesn't if it's in the test's data provider.

IIUC this is due to test discovery: when you have a @dataprovider enabled test, during the test discovery (regardless if you are testing a --group or a single file), PHPUnit calls the dataprovider to build the matrix of the actual tests that have to be run (you see that nicely in the failure of the PhpUnitCliTest), than uses the matrix to run each test individually with its own resolved dataset. Since this occurs before the test execution, if there are @trigger_error thrown, they're not yet captured by the phpunit-bridge mechanism.

Since this occurs before the test execution, if there are @trigger_error thrown, they're not yet captured by the phpunit-bridge mechanism.

Yeah that sounds like what is happening. This might need upstream work. I'll try to have a look at why our deprecation skipping is not working in these instances.

Done some more thinking about this problem. It's because our skipping deprecation error handler is only registered by a test listener - which occurs after the providers have run. I think the best solution for this is https://github.com/symfony/symfony/pull/37733 - which will move the skipping of deprecations upstream.

@catch we already use Sf5 phpunit-bridge so we can move getSkippedDeprecations to a file. Just not got round to it.

I think then we can close this in favor of #3180042: Add baseline for deprecation testing.

Also set back priority to normal since apparently we worked around the reasons for bumping it to critical in #3, we are on both on PHP 8 and PHPUnit 9 more or less happily atm