Add a flag to set up test environment only once per test class; introduce setUpBeforeClass() and tearDownAfterClass() [PHPUnit]

PHPUnit has two methods, setUp() and setUpBeforeClass(). teardown has similar structure. So, a test author can choose to implement either. As a general rule, I agree that one drupal install per class is good.

Nice work so far!

#2 sounds like an excellent pattern to me.

Take care. The abuse of this method can drive you to coupling between tests and lack of isolation.

Hmm...I both like and the idea and am a little worried by it. The idea that a test class now only has to get setup once for all tests certainly seems pretty awesome if only for the speed bump. But at the same time (and maybe I am mistaken so someone could knock some sense into me), I can't help but think of a class now being one large continuous test. What @carlescliment said in isolating the issue (instead of reading through the setup and test function, you would now be reading through the setup + all test functions prior to and including the test function to debug what the test issue could be) does worry me a bit.

@sun, Thanks for the clarification. I was reading through it and the line:

It cannot be emphasized enough that sharing fixtures between tests reduces the value of the tests.

Was what brought up part of the doubt. But if we're not able to isolate the tests anyways (and this code moves us towards phpunit which hopefully does help with our test isolation issues and overall will do much more good anyways), then awesome :) I'll make some time and try to post some updates in your sandbox towards this.

@sun what about installing drupal normally in setUpBeforeClass and saving a database snapshot that we can restore on setUp? would that be a good compromise between speed and keeping things working?

we have discussed db restore in the past and Upal even implements that approach. But we are now using the Testing profile a lot more and we need to keep stripping that down so that it does not do menu build and node install and so on. I think we can speed up Testing and use more UnitTestCase such that we don't have to change to DB restore method.

Is there any issues regarding PHPUnit migration ? A [meta] issue or something that records every moves to PHPUnit ?

Ok so I created this [meta] issue: #1801176: Deploy a PHPUnit system with a bottom-up approach

I also contacted Moshe Weitzman to ask him if there is any work in progress that I can take over. I am so pissed-off by SimpleTest today that I am making my personal battle to work on replacing it by PHPUnit.

Added API changes.

Now that the test execution process has been revamped a bit I wonder whether we can't get away without the complexity that is added by the conditional logic in this patch.

I.e. couldn't we go back to something like #10 but without converting the whole slew of test classes:
- Make TestBase unconditionally call setUpBeforeClass() and tearDownAfterClass() at the appropriate places.
- Rename WebTestBase::setUp()/tearDown() to setUpBeforeClass() and tearDownAfterClass(). The same for DrupalUnitTestBase and UnitTestBase.
- Add empty setUp() and tearDown() methods to those classes.
- Convert one or two test classes to setUpBeforeClass()/tearDownAfterClass().

??
I may be missing something.

If we stick to this approach we should rename SingleEnvironmentInterface. In general, I'm not sure on the terminology. If we stick to environment the two methods really should be setUpEnvironment() and tearDownEnvironment().

Hmm... OK, that makes sense then. But the ultimate goal is still to not have this interface and to have all test classes implement this, right? (Even if we might not reach that goal for D8) And for e.g. new tests that get added they should also implement this right off the bat.

Regarding the method names, I didn't know PHPUnit supported that, that's cool. I still think the interface should be renamed to "SingleEnvironmentInterface" and it should have at least one or two more sentences of additional information:
* That implementing the interface leads to a performance increase.
* That all new test classes should implement this so that we can eventually remove the interface (given that that is true)
* That this is compliant with what PHPUnit does.

One nitpick:

+++ b/core/modules/simpletest/lib/Drupal/simpletest/TestBase.php
@@ -801,6 +801,72 @@ public function run(array $methods = array()) {
+        $implements_interface = in_array('Drupal\simpletest\SingleEnvInterface', class_implements($parent_class));

This could use is_subclass_of() I think.

As the logic is a little bit involved this could use a couple more eyes, although I couldn't find any flaws.

I'm not sure why this can't simply be another base class, where setUp() is marked final, and setUpOnce() gets called on subclasses.

How about adding a flag in getInfo()? This will have the advantage of being easy to backport, too.

I admit I have not read this issue, but I did cmd+F for "upstream" at least and didn't find anything mentioned.

I'd really rather not go down the same road we did with SimpleTest of essentially "forking" the library and making it full of Drupalisms, some of them actually useful (like this one sounds). Can we file an upstream patch for this feature?

@webchick setUpBeforeClass() and tearDownAfterClass() are already part of phpunit. This issue brings us *closer* to PHPUnit, not farther.

Rock! :) Carry on, then!

@webchick: I'd really rather not go down the same road we did with SimpleTest of essentially "forking" the library and making it full of Drupalisms,

Pretty sure that boat has sailed, unless the goal is to re-implement SimpleTest based on the real framework, in which case this issue needs to be waaaay postponed. https://packagist.org/packages/simpletest/simpletest

@moshe: setUpBeforeClass() and tearDownAfterClass() are already part of phpunit. This issue brings us *closer* to PHPUnit, not farther.

How to make Drupal do modern, performant functional testing using PHPUnit as a framework:

1) Decide that the whole thing can be dependency injected by saying $app = new DrupalKernel('test', $etc, $etc);

2) See: http://symfony.com/doc/current/book/testing.html#your-first-functional-test

Seems obvious to me this is not a goal for Drupal 8 or it would have happened a while back. Like, first. :-)

But as for this issue: Which do people prefer? Add an interface, or make a flag in getInfo()?

Patch no longer applies, but it is probably more work than a simple re-roll at this point since much has changed in the past year.

This should be resurrected - but should be split into separate issues for Kernel / Browser / JS tests.

It feels if we had fixed this issue in 2012 the Drupal Association could save tens of thousands of dollars on CI environments.

(Posted on wrong issue)

How often are we running these tests? Seems like a good investment of time to cut the execution time in half, even if it's a matter of allowing us to know earlier that the patch works.

It has been over a year since someone looked at it. If @chi's perception of this is right, there are thousands more in savings that have been lost (let alone the environmental impact of running millions of processes on servers as part of this).

@joachim’s 2023 suggestion was to split the problem into separate issues for each test type. I tried to summarize would this would look like using ChatGPT to highlight what this would look like, along with justification and next steps. I'm mostly doing this to nudge this ahead.

Do these new issues sound like good places to begin? Has there been other work to do this already?

⸻

1. Kernel Tests: Optimize setUp() Execution

Goal: Avoid redundant setUp() execution for each method in Kernel tests.

Justification:
• Kernel tests do not boot the full stack and have fewer side effects.
• A shared setUp() could save significant time across large test classes.

Suggested New Issue:

“Optimize PHPUnit Kernel test execution by avoiding repeated setUp() calls per method.”

Action Items:
• Benchmark time saved when sharing setUp() across methods.
• Propose an attribute or annotation to allow opting in/out per class.
• Ensure tearDown cleanup logic remains reliable.

⸻

2. Browser Tests (Functional + FunctionalJavascript): Improve Test Efficiency

Goal: Investigate safe ways to cache installed sites between test methods in browser-based test classes.

Justification:
• Browser tests are the slowest due to full Drupal installs and HTTP interactions.
• Current design reinstalls Drupal for every method—even within the same class.

Suggested New Issue:

“Enable shared Drupal site installations across Browser test methods to reduce test run time.”

Action Items:
• Explore integration with #2900208 for site install caching.
• Identify classes that could safely share a site environment.
• Flag test classes with dependencies on state or side effects to avoid breaking isolation.

⸻

3. JavaScript Tests: Evaluate Efficiency Gains from Shared setUp()

Goal: Identify whether JS tests (Nightwatch or others) could benefit from setup sharing, or if they already do.

Justification:
• JS test infrastructure is separate (Node/Nightwatch/WebDriver), but setup redundancy may still exist.
• Even if not using PHPUnit, parallels in redundant test bootstrapping may exist.

Suggested New Issue:

“Audit and optimize Nightwatch JS tests for redundant setup executions across test methods.”

Action Items:
• Confirm whether test fixtures or browser states are rebuilt between methods.
• Identify opportunities for fixture reuse or browser session persistence.

⸻

Cross-Issue Meta Task: Central Coordination and Tracking

Create a meta issue to link all three above:

“Optimize test performance by limiting redundant setUp() execution across test types.”

⸻

Justification for Splitting

Joachim’s 2023 comment was correct: different test types have different performance bottlenecks, state requirements, and risk profiles. Attempting to solve them all in one issue creates blockers and lack of clarity. Splitting enables:
• Focused benchmarking
• Different contributors to work in parallel
• Faster consensus and review
• Targeted test infrastructure changes

The chatgpt output isn't helpful.

#2900208: Improve performance of functional tests by caching Drupal installations had more recent work on it, but the performance gains there aren't clear.

#3416522: Add the ability to install multiple modules and only do a single container rebuild to ModuleInstaller significantly reduced setup costs for both functional and functional js tests (possibly kernel tests too although it would be a smaller proportion of the test time).

The bigger problem here is that a lot of tests rely on starting off with a fresh environment, so individual test methods would have to be rewritten to account for changes made in other methods. This means the new attribute would have to be opt-in, which means no performance gain until it's applied individually to each test and those tests are potentially refactored - it's a lot of work and it's possible to optimize individual tests to have less methods individually without these APIs by doing more in one method.

I did some work on trying to optimize some of the very slowest tests over the past couple of years, and you can see how much work it is per test:

#3462556: Consolidate two test methods in NumberFieldTest
#3460654: Merge test methods in FieldUIRouteTest for better performance
#3463351: Consolidate Umami performance tests
#3463456: Consolidate ckeditor5's FunctionalJavascript tests
There are other similar ones around that stalled. So finding the slowest tests and speeding them up is for me a more efficient use of time. The ability to do setup/teardown once for a test could be helpful for some of those, but it's only part of the work - and you can just make test methods protected and call them from one public method to get pretty much the same effect.

#3390193: Add a drupalGet() method to KernelTestBase is likely to have much more impact too.

#3462762: [meta] Core test suite performance isn't up-to-date but has more issue references.

Thanks @catch I appreciate the better update!

There are a few critical points that I believe deserve attention.

1. Need for fast tests across all kind of modules

Fast tests are essential not only for Drupal core but also for contributed and custom modules. In my projects, I often avoid using data providers in kernel tests with large datasets because they become prohibitively slow. This limitation is frustrating.

2. Outdated testing System

The current testing framework has seen little improvement since the PHPUnit migration. It has become a tangled mess of gigantic base classes, traits, and poorly documented environment variables. A deep refactoring is necessary, and, I think, we should consider building a new testing framework from scratch. This framework should provide:
• Factories for generating demo content.
• Comprehensive assertion helpers.
• A helper to install Drupal.
• Strict separation between tests and the testing framework. See #3067979: Exclude test files from release packages.

3. A streamlined way to install Drupal with specified parameters.

Obsolete install system blocks progress. While not directly related to testing performance, the outdated installation subsystem is a blocker for improving the testing framework. It still relies on global variables, legacy functions like install_tasks(), and patterns dating back to Drupal 6 (or earlier). The new testing framework needs a clean way to spin up isolated Drupal instances.

@chi yeah I also stopped using data providers for several tests in core like #3462561: Stop using a data provider in UserPasswordResetTest, agreed it's a pain and if you don't pay attention to test runtimes it can add minutes to runtimes.

I don't really see how e.g. KernelTestBase::setUp() can be implemented once per test class though, because it's setting a load of class properties. But if we were going to try to do this, then kernel tests are probably the right place to start because there is often less custom logic in setUp().

For me though I think I would consider making it a new base class like SingleEnvironmentKernelTestBase or similar so we don't have loads of logic branching.

Installer, yes it's bad, issues like #3136855: Refactor install_* database helpers into an install helper class and child issues of #2876001: Installerless Drupal need help.

To expand on #63. Actual unit tests are generally fine with dataproviders because there's limited things you can do in ::setUp().

Functional and functional js tests there are much less use cases for data providers, and at least for functional tests, they can be converted to kernel tests once #3390193: Add a drupalGet() method to KernelTestBase is available.

So if we have a SingleEnvironmentKernelTestBase, and also do #3390193: Add a drupalGet() method to KernelTestBase, and then convert quite a lot of core and contrib tests to use those two things, that should give a good idea if there are any gaps left.

> In my projects, I often avoid using data providers in kernel tests with large datasets because they become prohibitively slow.

Agreed, and for me data providers are also the key use case for this issue's functionality.

> It has become a tangled mess of gigantic base classes, traits, and poorly documented environment variables. A deep refactoring is necessary

Yes!!

But I think rebuilding from scratch would be insanely complex - there would be lots of functionality to discover and ensure we don't lose. I think a steady refactoring can work too -- there is an issue for standardizing which folders we put traits in, for example. Standardizing naming and locations of things would be a big first step.

Moving classes and traits around should be straightforward now we have #3502882: Add a classloader that can handle class moves but also that discussion needs its own issue. Probably as a child/related issue of #3067979: Exclude test files from release packages.