Add baseline for deprecation testing

symfony/phpunit-bridge 5.2.0 was released today

Removing all the entries

diff --git a/core/tests/Drupal/Tests/Listeners/DeprecationListenerTrait.php b/core/tests/Drupal/Tests/Listeners/DeprecationListenerTrait.php
index 2ef406c29d..7fcbff7132 100644
--- a/core/tests/Drupal/Tests/Listeners/DeprecationListenerTrait.php
+++ b/core/tests/Drupal/Tests/Listeners/DeprecationListenerTrait.php
@@ -100,46 +100,6 @@ public static function getSkippedDeprecations() {
     return [
       // The following deprecation message is skipped for testing purposes.
       '\Drupal\Tests\SkippedDeprecationTest deprecation',
-      // The following Symfony deprecations are introduced in the Symfony 4
-      // development cycle. They will need to be resolved prior to Symfony 5
-      // compatibility.
-      'The "Symfony\Component\HttpFoundation\File\MimeType\MimeTypeGuesser" class is deprecated since Symfony 4.3, use "Symfony\Component\Mime\MimeTypes" instead.',
-      'The "Drupal\Core\File\MimeType\MimeTypeGuesser" class implements "Symfony\Component\HttpFoundation\File\MimeType\MimeTypeGuesserInterface" that is deprecated since Symfony 4.3, use {@link MimeTypesInterface} instead.',
-      'The "Symfony\Component\HttpFoundation\File\MimeType\FileBinaryMimeTypeGuesser" class is deprecated since Symfony 4.3, use "Symfony\Component\Mime\FileBinaryMimeTypeGuesser" instead.',
-      'The "Symfony\Component\HttpFoundation\File\MimeType\FileinfoMimeTypeGuesser" class is deprecated since Symfony 4.3, use "Symfony\Component\Mime\FileinfoMimeTypeGuesser" instead.',
-      'The "Drupal\Tests\Core\DependencyInjection\Compiler\LegacyMimeTypeGuesser" class implements "Symfony\Component\HttpFoundation\File\MimeType\MimeTypeGuesserInterface" that is deprecated since Symfony 4.3, use {@link MimeTypesInterface} instead.',
-      'The "Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher::dispatch()" method will require a new "string|null $eventName" argument in the next major version of its interface "Symfony\Contracts\EventDispatcher\EventDispatcherInterface", not defining it is deprecated.',
-      'The "Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher::dispatch()" method will require a new "string|null $eventName" argument in the next major version of its parent class "Symfony\Contracts\EventDispatcher\EventDispatcherInterface", not defining it is deprecated.',
-      // The following deprecation is listed for Twig 2 compatibility when unit
-      // testing using \Symfony\Component\ErrorHandler\DebugClassLoader.
-      'The "Twig\Environment::getTemplateClass()" method is considered internal. It may change without further notice. You should not extend it from "Drupal\Core\Template\TwigEnvironment".',
-      '"Symfony\Component\DomCrawler\Crawler::text()" will normalize whitespaces by default in Symfony 5.0, set the second "$normalizeWhitespace" argument to false to retrieve the non-normalized version of the text.',
-      // PHPUnit 8.
-      "The \"PHPUnit\TextUI\ResultPrinter\" class is considered internal This class is not covered by the backward compatibility promise for PHPUnit. It may change without further notice. You should not use it from \"Drupal\Tests\Listeners\HtmlOutputPrinter\".",
-      "The \"Drupal\Tests\Listeners\DrupalListener\" class implements \"PHPUnit\Framework\TestListener\" that is deprecated Use the `TestHook` interfaces instead.",
-      "The \"Drupal\Tests\Listeners\DrupalListener\" class uses \"PHPUnit\Framework\TestListenerDefaultImplementation\" that is deprecated The `TestListener` interface is deprecated.",
-      "The \"PHPUnit\Framework\TestSuite\" class is considered internal This class is not covered by the backward compatibility promise for PHPUnit. It may change without further notice. You should not use it from \"Drupal\Tests\TestSuites\TestSuiteBase\".",
-      // Simpletest's legacy assertion methods.
-      'UiHelperTrait::drupalPostForm() is deprecated in drupal:9.1.0 and is removed from drupal:10.0.0. Use $this->submitForm() instead. See https://www.drupal.org/node/3168858',
-      'AssertLegacyTrait::assertEqual() is deprecated in drupal:8.0.0 and is removed from drupal:10.0.0. Use $this->assertEquals() instead. See https://www.drupal.org/node/3129738',
-      'AssertLegacyTrait::assertNotEqual() is deprecated in drupal:8.0.0 and is removed from drupal:10.0.0. Use $this->assertNotEquals() instead. See https://www.drupal.org/node/3129738',
-      'AssertLegacyTrait::assertIdentical() is deprecated in drupal:8.0.0 and is removed from drupal:10.0.0. Use $this->assertSame() instead. See https://www.drupal.org/node/3129738',
-      'AssertLegacyTrait::assertNotIdentical() is deprecated in drupal:8.0.0 and is removed from drupal:10.0.0. Use $this->assertNotSame() instead. See https://www.drupal.org/node/3129738',
-      'AssertLegacyTrait::assertText() is deprecated in drupal:8.2.0 and is removed from drupal:10.0.0. Use $this->assertSession()->responseContains() or $this->assertSession()->pageTextContains() instead. See https://www.drupal.org/node/3129738',
-      'AssertLegacyTrait::assertNoText() is deprecated in drupal:8.2.0 and is removed from drupal:10.0.0. Use $this->assertSession()->responseNotContains() or $this->assertSession()->pageTextNotContains() instead. See https://www.drupal.org/node/3129738',
-      'AssertLegacyTrait::assertRaw() is deprecated in drupal:8.2.0 and is removed from drupal:10.0.0. Use $this->assertSession()->responseContains() instead. See https://www.drupal.org/node/3129738',
-      'AssertLegacyTrait::assertNoRaw() is deprecated in drupal:8.2.0 and is removed from drupal:10.0.0. Use $this->assertSession()->responseNotContains() instead. See https://www.drupal.org/node/3129738',
-      // PHPUnit 9.
-      "The \"PHPUnit\TextUI\DefaultResultPrinter\" class is considered internal This class is not covered by the backward compatibility promise for PHPUnit. It may change without further notice. You should not use it from \"Drupal\Tests\Listeners\HtmlOutputPrinter\".",
-      'assertFileNotExists() is deprecated and will be removed in PHPUnit 10. Refactor your code to use assertFileDoesNotExist() instead.',
-      'assertRegExp() is deprecated and will be removed in PHPUnit 10. Refactor your code to use assertMatchesRegularExpression() instead.',
-      'assertNotRegExp() is deprecated and will be removed in PHPUnit 10. Refactor your code to use assertDoesNotMatchRegularExpression() instead.',
-      'assertDirectoryNotExists() is deprecated and will be removed in PHPUnit 10. Refactor your code to use assertDirectoryDoesNotExist() instead.',
-      'Support for using expectException() with PHPUnit\\Framework\\Error\\Warning is deprecated and will be removed in PHPUnit 10. Use expectWarning() instead.',
-      'Support for using expectException() with PHPUnit\\Framework\\Error\\Error is deprecated and will be removed in PHPUnit 10. Use expectError() instead.',
-      'assertDirectoryNotIsWritable() is deprecated and will be removed in PHPUnit 10. Refactor your code to use assertDirectoryIsNotWritable() instead.',
-      'assertFileNotIsWritable() is deprecated and will be removed in PHPUnit 10. Refactor your code to use assertFileIsNotWritable() instead.',
-      'The at() matcher has been deprecated. It will be removed in PHPUnit 10. Please refactor your test to not rely on the order in which methods are invoked.',
     ];
   }
 

and running the command

SIMPLETEST_DB={your_db_url} SYMFONY_DEPRECATIONS_HELPER='generateBaseline=TRUE&baselineFile=test.deprecation.json' vendor/phpunit/phpunit/phpunit -c core core/modules/action/tests/src/Kernel/Migrate/d6/MigrateActionsTest.php

you get a test.deprecation.json that contains

[
    {
        "location": "Drupal\\Tests\\Listeners\\DrupalListener::endTest",
        "message": "The \"PHPUnit\\TextUI\\DefaultResultPrinter\" class is considered internal This class is not covered by the backward compatibility promise for PHPUnit. It may change without further notice. You should not use it from \"Drupal\\Tests\\Listeners\\HtmlOutputPrinter\".",
        "count": 1
    },
    {
        "location": "Drupal\\Tests\\Listeners\\DrupalListener::endTest",
        "message": "The \"Drupal\\Tests\\Listeners\\DrupalListener\" class implements \"PHPUnit\\Framework\\TestListener\" that is deprecated Use the `TestHook` interfaces instead.",
        "count": 1
    },
    {
        "location": "Drupal\\Tests\\Listeners\\DrupalListener::endTest",
        "message": "The \"Drupal\\Tests\\Listeners\\DrupalListener\" class uses \"PHPUnit\\Framework\\TestListenerDefaultImplementation\" that is deprecated The `TestListener` interface is deprecated.",
        "count": 1
    },
    {
        "location": "Drupal\\Tests\\action\\Kernel\\Migrate\\d6\\MigrateActionsTest::testActions",
        "message": "The \"Symfony\\Component\\HttpFoundation\\File\\MimeType\\MimeTypeGuesser\" class is deprecated since Symfony 4.3, use \"Symfony\\Component\\Mime\\MimeTypes\" instead.",
        "count": 1
    },
    {
        "location": "Drupal\\Tests\\action\\Kernel\\Migrate\\d6\\MigrateActionsTest::testActions",
        "message": "The \"Symfony\\Component\\HttpFoundation\\File\\MimeType\\FileBinaryMimeTypeGuesser\" class is deprecated since Symfony 4.3, use \"Symfony\\Component\\Mime\\FileBinaryMimeTypeGuesser\" instead.",
        "count": 1
    },
    {
        "location": "Drupal\\Tests\\action\\Kernel\\Migrate\\d6\\MigrateActionsTest::testActions",
        "message": "The \"Symfony\\Component\\HttpFoundation\\File\\MimeType\\FileinfoMimeTypeGuesser\" class is deprecated since Symfony 4.3, use \"Symfony\\Component\\Mime\\FileinfoMimeTypeGuesser\" instead.",
        "count": 1
    },
    {
        "location": "Drupal\\Tests\\action\\Kernel\\Migrate\\d6\\MigrateActionsTest::testActions",
        "message": "AssertLegacyTrait::assertIdentical() is deprecated in drupal:8.0.0 and is removed from drupal:10.0.0. Use $this->assertSame() instead. See https://www.drupal.org/node/3129738",
        "count": 40
    }
]

that is, all the deprecation triggered that were previously silenced now get summarised in the file.

Brainstorming: what we need to do here (and it's quite a lot) is to be able to produce a file for each test run (because we cannot have a single file, run-tests.sh does parallelise the test execution in multiple processes), and consolidate the files into one with the summary at the end of the complete run. That file will be the 'entire' baseline for Drupal. Then, that baseline should be generated with a special test run on DrupalCI that would update the deprecation baseline every (?), and be included in the repo (?)

During the normal test runs, the 'baseline file' should be unpacked and re-allocated for each test run so that the single test baseline could be checked against its run results.

Moving to MR workflow and bumped to D10

The baseline has 1000s of entries related to the usage of legacy Symfony 3 MimeType classes, which I suppose would go away once #3197482: Update Drupal 10 to depend on Symfony 5.4 (as a stepping stone to Symfony 6, for deprecation checking support) is in.

However, this makes me think whether we should have somewhere a list of deprecation patterns to be ignored while building the baseline as well as while comparing against it, a la PHPStan. That’s probably something for upstream, though.

Also, now the baseline is split per test suite following the logic DrupalCI (unit, kernel, build, etc), so in any case we will need to manually merge it on a first run, unless an infra job will be able to do it.

Rerolled after commit of #3197482: Update Drupal 10 to depend on Symfony 5.4 (as a stepping stone to Symfony 6, for deprecation checking support)

MR 1655 is about GENERATING the baseline

MR 1660 is about USING the baseline

Attached, the baseline that is resulting by 'merging' manually the files under

artifacts/run_tests.{test-suite}/phpunit-xml/deprecation_baseline.json

for each of the DrupalCI testsuite runs (build, functional, javascript, kernel, phpunit).

There are lots of deprecations thrown in the listeners like

    {
        "location": "Drupal\\Tests\\Listeners\\DrupalListener::endTest",
        "message": "The \"PHPUnit\\TextUI\\DefaultResultPrinter\" class is considered internal This class is not covered by the backward compatibility promise for PHPUnit. It may change without further notice. You should not use it from \"Drupal\\Tests\\Listeners\\HtmlOutputPrinter\".",
        "count": 1
    },

that would trigger for each single test... another reason for an 'ignoreFile' per #27.

I have filed upstream PHPUnitBridge - add a deprecation 'ignoreFile option' to allow silencing deprecation patterns

If we had #35 from upstream, then I think we would no longer need to bypass PHPUnitBridge's deprecation management, so we would not longer need a deprecation listener (or whatever it will be in PHPUnit 10). At the same time we could allow contrib to alter the ignore file, so not needing #3143388: Allow projects to add to the list of skipped deprecations either

Rerolled

This will become much simpler once #3272779: [Symfony 6.1] Replace deprecationListenerTrait with PHPUnitBridge ignoreFile will have been done.

MR 2372 is built on top of #3272779: [Symfony 6.1] Replace deprecationListenerTrait with PHPUnitBridge ignoreFile, and modifies run-tests.sh script only, so that baseline consolidation from different files occurs at the end of each run-tests.sh execution. It does not interact with Symfony's internals.

Working on this.

After #3272779: [Symfony 6.1] Replace deprecationListenerTrait with PHPUnitBridge ignoreFile, this is now rather simple.

MR 1660 --- USING the baseline

This is rather trivial. You have a baseline file --> just indicate it in the PHPUnitBridge configuration and you're done.

MR 2372 --- GENERATING the baseline

Just sligthly more complicated:

• run-tests.sh executes parallel test runs, each of which generating a baseline file of its own
• at the end of each test suite run on DrupalCI, the individual files are consolidated first in a suite baseline file, and then each suite file into a full baseline file
• at the end of the DrupalCI test run, you find the generated full baseline file in the test run artifacts at artifacts/run_tests.javascript/phpunit-xml/full_deprecation_baseline.json
• then you can update the baseline file for usage in core/.deprecation-baseline.json (see MR 1660 above)

The automated baseline generation includes also deprecations triggered from within tests marked @legacy, which does not make much sense.

Opened issue upstream, https://github.com/symfony/symfony/issues/46710

PR merged upstream, let's wait for a Symfony patch release update.

For an approach when to use ignoreFile and when a baselineFile, https://drupal.slack.com/archives/C1BMUQ9U6/p1655711760320739

Let’s include that in a CR

Upstram fixed. Now postponed on #3292730: Update to Symfony 6.1.2.

Squashing together MR 2372 and #2664570: Move Attribute classes under Drupal\Component to see what baseline juice we get

OK, now test the baseline generated by #53 (available at https://dispatcher.drupalci.org/job/drupal_patches/136993/artifact/jenki...) with MR 1660.

Well, we need to include #2664570: Move Attribute classes under Drupal\Component too to make it meaningful.

Suppress commit checks for the purpose of testing

So, what the experiments in #56 and #59 tell us:

a) the concept of deprecation baseline is different from the PHPStan baseline. It's more a 'watermark': while in PHPStan if you correct an issue without cleaning up the baseline you will get an error reporting that PHPStan was expecting an error that did not show up, this is not happening for this baseline. You can have less deprecations occurring in a test, and no error shown. But you can't have more than those in the baseline.

b) for the case of #2664570: Move Attribute classes under Drupal\Component, the generated baseline is huge, and on a run with the baseline applied we have more errors showing up --> probably it will be fair to add a ignore pattern in the ignoreFile in this case instead of baselining.

Back to NR since MR 1660 is practically the scope that needs to be done here based on the issue title - the baseline generation should probably be a separate issue.

#65 about ignoring deprecations, #3272779: [Symfony 6.1] Replace deprecationListenerTrait with PHPUnitBridge ignoreFile and https://www.drupal.org/node/3285162 could be better friends :)

Related, #3295618: Get PHPStan baseline generation via DrupalCI. IIUC the idea is that we should not use DrupalCI to generate baselines, either there or here - rather wait for availability of an infra that could allow that more easily.

PHPUnit bridge will not have a PHPUnit 10 compatible version. Deprecation baseline is now available from PHPUnit itself, but that would be a different story.