Tests are no longer modifiable

Could you clarify what you're trying to do and why you think this is needed, please?

The whole point of #2171683: Remove all Simpletest overrides and rely on native multi-site functionality instead was to remove each and every possible thing that could possibly leak from the test runner into tests.

All test results are muddied and inherently unreliable and invalid, if the same test does not produce the exact same result in all environments.

By including a custom settings.php instead of the empty default.settings.php, that is no longer guaranteed. Instead, there's a good chance for tests to fail out of nowhere on my machine, but "strangely" not on anyone else's machine. — Cause: settings.php.

I want to add more modules to the tests without needing to extend every single test

I don't really understand what this means?

1. How can you add modules via settings.php?
2. Why don't you use a base test class like all other tests in core that don't want to repeat themselves? That's why we introduced public static $modules...?

+++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
@@ -808,6 +808,10 @@ protected function setUp() {
+    if (file_exists(DRUPAL_ROOT . '/sites/default/settngs.testing.php')) {

settings is misspelled.

This is not a regression, since a feature like this was never supported in the past.

Just because something was possible doesn't imply that it was supported and intended to be used and work that way. I hope we can agree on that principle.

Architecturally, this feature is asking for a mechanism that is similar to our existing Database driver negotiation — i.e., instead of hard-coding a particular driver, the system uses an available driver.

However, that mechanism does not exist for various storage concepts in core yet. Instead, the application hard-codes a default storage; which is a database based implementation in almost all cases.

One possible mechanism that is not only used by Symfony but also common practice in other languages and applications, is the concept of allowing $_ENV to override application-level configuration.

@msonnabaum mentioned this possibility a few times already; most recently in #2226761: Change all default settings and config to fast/safe production values

Whether the concept of introducing $_ENV (pretty much as in Symfony) would work for this use-case mostly depends on how complex those overrides are — i.e., whether they are just some static declarations (simply overriding service container definitions), or whether they involve dynamic PHP code.

→ A concrete example of what such a custom settings file actually contains would help to understand the actual requirements.

Based on that, we can check how this can be implemented in the most sensible way.

Thanks for your feedback.

1. We are aware of the available bootstrap service container override options. They do not apply to tests.

2. The test environment is isolated on purpose. We've been working very long and hard to fix all the bugs in which the test runner environment leaked into tests. We also worked hard to ensure that tests yield the same results in all environments. Due to that, what you're reporting is the opposite of a regression; the test environment is intentionally not affected by the test runner's environment.

3. It is still not clear whether that custom config storage override snippet presents all of the intended/necessary overrides to enable this feature. For example, why is that limited to overriding the bootstrap config storage, but does not change the active storage? Also, I assume that some storage engines will require connection credentials in order for this to work?

4. Perhaps it's a bit early before having seen the full set of overrides, but TBH, just the given example settings.php snippet in #14 alone rather looks like a dirty hack and not something we'd want to support officially. I like the general idea of this feature request, but we should think some more about how this can be implemented cleanly.

It's too early to comment on the patch, but FWIW, the storage has to be overridden before the installer is invoked, not afterwards (as in the latest patch). Otherwise, Drupal would be installed using the default storage engine(s), instead of the custom, environment-specific one(s). The installer itself is an integral part of tests and involves plenty of storage operations, so custom storage engine overrides need to be applied already.

Sorry, I believe that my comments were phrased in a collaborative, constructive, and productive way, but yet, the communication here creates an environment and atmosphere that isn't fun and that I don't want to participate in.

Happy to re-review this patch once a proper solution has been found for the actual feature request (which makes sense to me, given the amount of swappable storage engines in D8).

For what it's worth, I also think the issue summary needs to be rewritten to explain the actual use-case and intention, which was only vaguely described in comments later on. Likewise, a complete concrete example of intended service overrides would certainly help everyone to figure out the right architectural solution.

Thanks for contributing to Drupal.

• $test_class appending confused me. Do we *really* need this variable? I would suggest to remove it, or at least adding an inline comment explaining the addition.
• The possibility of adding a settings.testing.php file should be documented somewhere, not sure where exactly.

All right. Tried this out locally and it works beautifully.

Looked at the code again and with the added code comments I have nothing to complain about.

Like @penyaskito I thought this feature should be documented somewhere at first, but I really don't know where. I thought about adding an empty settings.testing.php to the repository and adding some documentation to that, but that would just be noise for 99% of people who don't need to care about this. So let's just do this, maybe we can find a better way to document this in the future.

I do really think we should do this, though. The issue summary lists a very good example for which you can use this. @chx wanting to swap storage controllers in all tests is a IMO 100% legit other use-case. I imagine people writing custom storage backends and also cache backends will probably want to do that in testing as well. @sun has a similar use-case in #2161591-53: Change default active config from file storage to DB storage. So yeah, let's just do this.

Re-read some of @sun's comments above. Some notes.

- A concrete example of how such a settings.testing.php file could look like is given in the issue summary. I thought of the use-case of swapping a different cache backend, which you can actually do in settings.php as well, fairly easily.
- The problem of reduced reliability is valid, but I don't think it's a killer for this feature. I.e. if something fails on one machine but not on another it's very trivial to figure out whether settings.testing.php was the cause, as you just need to look whether that file exists.
- I agree that we need some less hacky mechanism to swap things. I.e. right now it's pretty hard to swap out the storage controller of an entity type from settings.php. That's why @chx had to swap the config storage. That's not tangential to this issue, though. If there is some easier swappability mechanism I still would like to have the possibility of swapping it just for tests.
- I don't really care whether this is a regression or not. Let's just do this?

In other words I don't really understand the fundamental reservations @sun has against this. I see "there is no use-case for this" and "this mitigates test isolation" as points, but a) I think there are some use-cases that have been explained here and b) the reduced isolation is very minimal and also can be wanted, in the same use-cases that have been mentioned.

1. +++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
   @@ -808,6 +809,21 @@ protected function setUp() {
        install_drupal($parameters);
   +    if (file_exists(DRUPAL_ROOT . '/sites/default/settings.testing.php')) {
   

   Potential storage service overrides should still apply to the installer already, because the installer pretty much writes out the entire system.

   I.e., the earlier patches in here were correct, only the later ones started to ignore the installer.

2. +      copy(DRUPAL_ROOT . '/sites/default/settings.testing.php', $directory . '/settings.testing.php');
   

   As mentioned before, I generally support the idea of this feature request. — However, what makes me uncomfortable with the proposed solution is that executing a custom PHP file with arbitrary contents yields more or less the exact situation that triggered this issue:

   Instead of supporting a particular feature through a well-defined and explicit mechanism, people can create "bug reports" and claim that some arbitrary custom override is supposed to work.

   Can we limit this to a test.services.yml file?

   That would at least minimize the scope to overrides for the service container only. In addition, it would also no longer execute random PHP code, but a static/declarative definition.

3. This patch only touches WebTestBase, which is confusing.

   Shouldn't the identical mechanism also work for DrupalUnitTestBase?

   The vast majority of storage related tests are DUTB [or will become so] — web tests ought to be limited to testing UI operations only.

4. +++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
   @@ -808,6 +809,21 @@ protected function setUp() {
   +      file_put_contents($directory . '/settings.php', "\n\$test_class = '" . get_class($this) ."';\n" . 'include DRUPAL_ROOT . \'/\' . $conf_path . \'/settings.testing.php\';' ."\n", FILE_APPEND);
   

   I do not understand the $test_class addition here. Seems unrelated?

Thanks!

Just to be clear (again), I do support to add this feature in general, as also outlined much more in-depth below.

But yes, pretty much in line with the entire rest of the D8 architecture, facilities like this really ought to be explicitly and officially supported in a well-defined way.

That is, because it doesn't really help anyone if exactly one person figured out how to hack the system in order to inject a custom storage engine instead of the default one(s) into tests, whereas (1) the maintainers themselves have no idea of what exactly those hacks entail, (2) the testing framework undergoes continuous changes and no one knows whether those might break some undefined use-cases, and (3) everyone else having the same use-case will have to figure out and apply the exact same or similar hacks via custom, unreliable PHP code that doesn't meet any actual feature specification on their own.

Just in terms of product/maintenance aspects alone, that is the very definition of horror. ;-)

Therefore, the intention always was and still is to seek for a "proper" solution that (1) meets and resolves a concrete use-case and (2) actually can be officially supported for realz.

Now, if you could avoid the inappropriate personal attacks (and whichever "fight" you assume to have with anyone) in your communication, then I'd be happy to continue to work with you to make this happen, because I think it's a good feature to have in case it's well-scoped and well-defined. - But to be clear, your communication here thus far does not fly; if you continue to do that, then I'm out of this issue.

1. re: WebTestBase vs. DUTB:

   Here's where I'm coming from: Having a look at a typical class hierarchy of e.g. FieldUnitTestBase (DUTB) — all of those tests are entity/field storage related.

   My assumption is that it's pretty much exactly all of those storage related tests that you are actually interested in, in case you have the use-case of running all tests against a custom storage engine.

   The storage related services actually get injected into many other services (Properly Decoupled Spaghetti++), so in almost all cases, it is sufficient to replace the storage service with a different implementation, and all of the dependent tests will actually use it.

   As mentioned previously, web tests only exist to test for UI-level aspects; e.g., that clicking link Foo leads to page Bar, and pressing form button X yields Y... — Aside from a couple of very old legacy web tests that were not converted/separated yet, I'm not actually able to see the point of running web tests against a custom storage engine...?

   Don't get me wrong, I think it's fine to include web tests, but that would primarily happen for consistency only — the real meat of storage related tests (thankfully) are not really web tests anymore (in D8).

   This means config, entity, field, key/value, cache, etc.pp... it's completely pointless to use WebTestBase for storage related tests, and unless I'm terribly mistaken, the vast majority of these tests is using DrupalUnitTestBase in D8 already.

   DrupalUnitTestBase uses DrupalKernel, the consumption/registration of additional service providers happens within DrupalKernel, so additional service providers are consumed by DUTB, too.

2. It's mainly the aforementioned considerations that lead me to the conclusion that we are actually seeking for a (officially supported) mechanism for replacing/overriding storage engines with custom ones in tests, in a well-defined and controlled way.

   I'd expect that to work with e.g. a test.services.yml service provider file in the site directory, which essentially replaces/overrides all the service definitions from core.services.yml of interest, so that they point to implementations of a custom storage engine instead; for example:

   services:
     keyvalue:
       class: Drupal\mongo\KeyValueStore\MongoDbKeyValueFactory
   

   I'm not sure why you're saying that the YAML service provider approach cannot be used for replacing service definitions (as that is one of its only two purposes), so if there is a bug with that (possibly the merging algorithm?), then we should fix that.

   An approach like this would make at least me much more comfortable with this feature addition. Because it means that we have a crystal-clear way for overriding default storage services in tests. By removing the randomness of completely custom executed PHP code, the feature can be officially supported (which is also in your interest).

   In fact, a well thought-out solution here will also allow Drupal 8 core itself to improve its test coverage through minimally customized PIFR environments. This happens to be an actively discussed topic currently, whereas arguments of others even go up to the extent of removing all code from core that is not fully covered by tests — whereas alternate implementations are only not tested, because there is no easy + officially supported way to inject them into the testing environment.

   For example, #2160433: [policy, no patch] Move PostgreSQL driver support into contrib

   That is pretty much the main reason for why I'm in support for the general, underlying idea of this feature request.

   And as a very concrete additional example, an officially supported feature like this would allow us to battle-test the entire core test suite against #2208617: Add key value entity storage (which apparently presents the canonical abstract implementation for custom storage engines)

3. I just noticed that the patch hard-codes sites/default as the test-runner/parent site directory.

   sites/default is not guaranteed to be the site directory of the test-runner/parent-site.

   Perhaps (much) more interestingly, by consulting the proper parent site, the facility of applying custom storage service overrides can actually be tested through an automated test.

   I.e., a recursive web test like SimpleTestTest, but for the sake of core developer sanity, let's please create a separate test class for this new test... ;-) — With above mentioned support for DUTB, the test executed in the child site can actually just be a DUTB test (improving test performance).

   We do have quite a range of alternative implementations for various services in core already, so testing an actual custom service override should be possible. (Above mentioned k/v entity storage would make most sense from my perspective, since it actually maps 1:1 to the actual use-case, but yeah, that hasn't landed yet...)

   Test coverage == clearly defined expectations == supported feature

   (Just to be clear: Executing random PHP code does not meet the requirement of that equation.)

4. re: $test_class:

   I've read the issue summary, but all it states is this:

   a $test_class variable tells it which test it runs.

   Doesn't explain how that is useful? The test class is always reported by the test runner already. Why does the system under test actively need to know the test class name?

OK, I'll take another stab at providing reason why I think the current approach is feasible and correct.

Mind you that I thought long and hard about this and re-read each of @sun's comments multiple times. As @sun's mentee I tend to agree on a lot of things with him and I feel that often when he shares architectural concerns with an otherwise agreed upon patch it's for the better. In this issue, I cannot come around disagreeing, though.

WebTestBase vs. DrupalUnitTestBase

As DrupalUnitTestBase does not do real unit tests and - as @sun points out - a lot of DrupalUnitTestBase-tests do storage-related things they would lend themselves extremely well for the swappability that this issue wants to introduce.

So I think we all agree that in the end we want to provide a way of overriding / swapping things for both WebTestBase and DrupalUnitTestBase.

However, the question on whether to focus this issue primarily on WebTestBase or on DrupalUnitTestBase is tied to the general architectural decision of whether to use a settings.testing.php file or not in a chicken-and-egg manner:

- If you - like @sun does - have reservations against the settings.testing.php file anyway and think that the patch needs to be re-architected anyway it seems natural to introduce the re-architected code in TestBase, so that both DrupalUnitTestBase and WebTestBase can utilize it, solving the duplicity problem nicely and elegantly.

If - like @chx and I do - believe, however, that the settings.testing.php approach is correct then it makes sense to introduce this behavior for WebTestBase only for now because that is the only type of test that currently has a settings.php file in the first place. We could then - in a follow-up - explore creating a settings.php file for DrupalUnitTestBase-tests as well. This would make sense regardless of this issue (... in my opinion at least. Let's not bikeshed this detail here, though, please.)

So the question on whether or not DrupalUnitTestBase should receive a similar treatment to WebTestBase should be postponed on resolving the other, larger question. Let's please just leave it out entirely for now.

settings.testing.php vs. explicit service overrides

In theory explicit overrides are much preferable over including a PHP file which can run arbitrary code, I think there is no question about that. The question is rather how feasible it is that we can provide an explicit mechanism for overriding all of the things that we can override.

Service overrides

@sun discussed service overrides above, so let's look at those in detail first. @sun mentions the possibility of simply specifying overrides in a YAML file. Looking at DrupalKernel that should actually work, but I'm pretty sure that literally noone is aware of that "feature". It is in the same way "not official" - i.e. not documented, not explicitly supported, not tested, etc. - as the settings.php overrides for tests are claimed to have been "not official". Proof of this is (among other things) LanguageServiceProvider. Specifically the overriding of the language manager, which could in theory (given that that actually works) be // Add site specific or test service providers.
easily done in language.services.yml. This override is often used as the canonical example of explaining to Drupal 7 developers why the whole "service" concept makes sense. Thus, I can say with certainty that it is using the "official" way of overriding a service. Therefore to not convolute this issue even further this point must be explored in a separate issue. Only when it is proven that this actually works, a testing-specific services.yml can be discussed. For now we must assume that a separate ServiceProvider class is needed for that. Note that this is only a minor point, because we need some sort of discovery mechanism (see below) either way: either for the service provider or for the YAML file.

In theory it would not be a problem to allow to put a TestingServiceProvider class in /sites/simpletest which gets picked up when building the test container. In theory for two reasons:

1. If implementing this correctly, we would have to create a separate TestingKernel which would pick this class up. I don't think adding a drupal_valid_test_ua() check to DrupalKernel is going to fly (of course I don't get to decide that, but I sincerely hope that that wouldn't fly!). That however would require to establish an entire separate bootstrap process for tests. That might become possible with #2016629: Refactor bootstrap to better utilize the kernel but TBH a pipe-dream at this point.
2. We already have a facility to allow to:
   

   // Add site specific or test service providers.

   (DrupalKernel, l. 238) but that requires setting global $conf (yes, it's still $conf...) somewhere. Where is global $conf intended to be set? In settings.php, of course. When reading that particular part of DrupalKernel the current approach of a test-specific settings.php feels quite natural.

   Note: Let's please not discuss the fact whether this issue is a regression or not any more. This issue is already very loaded without that detail. I realize that the quoted code comment could be seen as an argument for one side in that fight. But again, please let's ignore that detail.

In conclusion:
There could be cleaner ways of providing one-off service providers, for a custom site or for tests. And hopefully we will get there to enable that eventually. But as it stands, the one that is used in core is by setting a global in a settings.php file.

Settings overrides

The example provided by @chx uses a Settings override to specify the bootstrap config storage, in order to override the module list. Although that is very hacky, it is a working (I haven't tried it, but I believe @chx if he says it works) and probably still the cleanest way of achieving that. In general I believe it makes sense to allow for settings overrides. But at the very least until someone provides a different way of achieving the same result I think we must support this.

So how do we override Settings? With a settings.php file, of course. It's really just that. I've heard @dawehner talk about moving Settings declaration into YAML, but - again - that's just a pipe dream at this point. And when you include a PHP file, there is no middleground: either you include it or you don't, but if you include it you can't prevent people doing all sorts of crazy things in there. I.e. there is no way to have a (PHP) file that's just for overriding settings.

Override of globals

@chx mentions the potential need to override $base_url for tests. I can't claim that I fully understand that use-case but it hasn't been refuted so far, so I can only assume it's a valid use-case that needs to be supported. This is - again - a situation where we could certainly do better. I.e. we shouldn't be relying on globals at all at this point. But the fact of the matter ist that we do, and - for now - we need to support that. Here the argument is the same as for the settings overrides. Globals are overriden in a PHP file that is included (== 'settings.php'), end of story.

The conclusion is similar to the one for service overrides already: We have a system for overriding various parts of the system in a one-off fashion. That system is settings.php. It could be improved and replaced in parts or entirely to make for a much cleaner system. And everyone would applaud that, I think. But currently that is the system that we have. So now that we have a use-case for overriding various parts of Drupal in a one-off fashion - namely for tests - we should use that system. KISS.

Test isolation

Surely the settings.testing.php would by its nature reduce isolation and reliability of tests. @sun, you mention "developer sanity" being at stake and that debugging would be greatly impaired by this change. If thought about this for a long time, but I could not come up with a single reason why that would be the case. To maybe make it easier to find the specific point of disagreement, I will state some things that I would consider consentual, but in aggregate they would dispute your claim. Therefore I assume that you will disagree with at least one of those points:

1. Many developers will not ever see a settings.testing.php file: It is primarily purposed for testers (i.e. writers) of alternate storage backends / DB drivers / cache backends / ... Neither in core nor in contrib more than a few percent of people will use this.
2. A settings.testing.php file will generally be single-purpose: It will be used to harcode-enable the MongoDB modules in tests or to store config with the key-value store (or whatever) but not both. While that's absolutely possible it doesn't make much sense to include various source of failures into one environment.
3. If you do encounter a test failure that someone else cannot reproduce it will take literally about 10 seconds (at most) to find out whether you have a settings.testing.php file on your system. If you do have one, just (temporarily) remove it, and re-run the test. If your test now passes you have the reason for your failure. If not, there must be another issue. In other words: Specifically the issue of impaired debugging I fail to see conmpletely.

Misc

Let's please leave the following items out of the discussion completely. They are implementation details and can be discussed once we have agreed on a general solution.

• Whether or not to provide a $test_class in settings.testing.php
• Whether to take the settings.testing.php from /sites/default or from $conf_path

Let's please leave the discussion about this completely out of this for the sake of not loading this issue even more. I'm pretty sure that @chx would be happy to remove that bit of code if there were no other objections.

TL;DR
We need something to improve our code quality by enhancing our test dimensions, though more documentation on both the new settings.testing.php
and existing override mechanism should get some focus.

In an ideal world we would not only have proper abstractions and layers
on the code side but we would have similar concepts on the tests.

Pure unit tests aka. UnitTestBase

For them you don't have container or whatever, so go along there is nothing to see here.

Code integration tests aka. DrupalUnitTestBase

These tests just rely on the DB + enviroment (kernel etc.) so you want to ensure
that your alternative storage backend still works on the pure API side. Therefore
you most of the time "just" need a way to swap out single services. YML files seems
to be totally okay for that. These tests most of the time don't deal with a
request object so for example changing the base url is not really needed. For everything else using the global is currently deprecated anyway but you would use a different request context service which does some fancy things. This should be entirely possible using yml files/container passes.

One problem could be that the DUTB tests rely on storage details, which aren't part
of the public API. Being able to swap out the underlying storage would help us to
cover those problems.

Webtests

These tests rely on a fully functional drupal so much like people do have a settings.php to change stuff on
their site we need to have test coverage for such chanages. Having a settings.test.php seems to be a pragmatic approach for that. I think we all agree that we should move away from the globals, try to avoid settings as they are a sign of work before bootstrap etc.
I really wonder whether we want to provide some default testing settings php which explains HOW you override
specific things in the right way.

On top of that I am really looking forward to being able to use drupal with composer which would allow you
to have potentialy control over index.php and a maybe site specific drupal kernel at the end of the day.

+++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
@@ -799,6 +800,13 @@ protected function setUp() {
+    if (file_exists(DRUPAL_ROOT . '/sites/default/settings.testing.php')) {

Just in general: If you do create this file you will be aware of the problems you might have, so if stuff fails you will know that you might first run core only tests first.

@chx promised to improve the documentation in the future, even I would like to see something in Drupal itself, but well you can't have just the cherries.

+++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
@@ -775,7 +775,8 @@ protected function setUp() {
+    copy(DRUPAL_ROOT . '/sites/default/default.settings.php', $directory . '/settings.php');

@@ -799,6 +800,13 @@ protected function setUp() {
+    if (file_exists(DRUPAL_ROOT . '/sites/default/settings.testing.php')) {

The difference between these two is that there is only one default.settings.php file in Drupal and that happens to be located in /sites/default/default.settings.php.

The installer always copies the files from that location. There is no other source.

Hence the "off-topic" reference to #1757536: Move settings.php to /settings directory, fold sites.php into settings.php, which (finally) moves this file into /core/default.settings.php.

Tests are not necessarily executed from within the default site.

Therefore, it is wrong to check the default site directory instead of the actual site directory of the test runner.

Test coverage?

It seems like this is still under discussion.

This will need an update after #2241633: Simplify site-specific service overrides. I.e. we will want to allow a test-specific testing.services.yml as well.

Will try to work on this this week.

Here we go. Added support for a testing.services.yml*. And also added test coverage for that.

I also slightly expanded the existing test coverage to also test the $test_class local variable. I verified that removing that actually results in fails in SimpleTestTest.

* Even though we have a settings.testing.php - which is consistent with settings.local.php - I called this testing.services.yml to be consistent with core.services.yml. This has the result that we have settings.testing.php and testing.services.yml right next to each other which seems inconsistent, but anything different would be (even more) mind-boggling to my mind.

1. +++ b/core/modules/simpletest/lib/Drupal/simpletest/TestBase.php
   @@ -54,6 +54,13 @@
   +   * The site directory of the original parent site.
   +   *
   +   * @var string
   +   */
   +  protected $originalSite;
   +
   

   was that previously just not documented? I don't see whether this is set.

2. +++ b/core/modules/simpletest/lib/Drupal/simpletest/Tests/SimpleTestTest.php
   @@ -237,7 +272,7 @@ function confirmStubTestResults() {
    
   -    $this->assertEqual('6 passes, 5 fails, 2 exceptions, 1 debug message', $this->childTestResults['summary']);
   +    $this->assertEqual('10 passes, 5 fails, 2 exceptions, 1 debug message', $this->childTestResults['summary']);
   

   Can we put something like a @see to the part of the code which produces the actual assertions? stubTest()

3. +++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
   @@ -843,6 +844,21 @@ protected function setUp() {
    
   +    // Allow for test-specific overrides.
   +    $settings_testing_file = DRUPAL_ROOT . '/' . $this->originalSite . '/settings.testing.php';
   +    if (file_exists($settings_testing_file)) {
   +      // Copy the testing-specific overrides in place.
   +      copy($settings_testing_file, $directory . '/settings.testing.php');
   +      // Add the name of the testing class to settings.php and include the
   +      // testing specific overrides
   +      file_put_contents($directory . '/settings.php', "\n\$test_class = '" . get_class($this) ."';\n" . 'include DRUPAL_ROOT . \'/\' . $conf_path . \'/settings.testing.php\';' ."\n", FILE_APPEND);
   +    }
   +    $settings_services_file = DRUPAL_ROOT . '/' . $this->originalSite . '/testing.services.yml';
   +    if (file_exists($settings_services_file)) {
   +      // Copy the testing-specific service overrides in place.
   +      copy($settings_services_file, $directory . '/services.yml');
   +    }
   

   DUTB tests could also profit at least from the custom testing.services.yml file.

@chx
Thank you for the pointer. I don't have tools available to look it up at the moment.

Well, tobias agreed that it makes sense to support DUTB as well.

OK, here we go. Added a bunch of inline comments to SimpleTestTest. Also added the same hunk of code to DrupalUnitTestBase.

I wanted to write a test for that, but we don't have a sort of SimpleTestTest-Inception-Like Test for DUTB, so that's not really possible. I did verify that the overrides do work for DUTB tests, however.

@dawehner: RTBC? :-)

See #2262215: WebTestBase and TestBase properties not declared or documented. for the $originalSite part.

52: 2229011-51-test-overrides.patch queued for re-testing.

Conflicted on #2259275: Rename "DrupalUnitTestBase" to KernelTestBase.

Thank you for adding the explanations!

+++ b/core/modules/simpletest/lib/Drupal/simpletest/Tests/SimpleTestTest.php
@@ -184,23 +216,41 @@ function testWebTestRunner() {
+    // This causes

... what ?

Here we go. This fixes the test comments.

Two interdiffs because I merged 8.x in between. Also, if you - like me - fail at math: "5th to 9th" is actually 5 things, not 4...

Whether this is a problem of people or of the english language itself is not clear for me.

+++ b/core/modules/simpletest/src/Tests/SimpleTestTest.php
@@ -184,23 +216,42 @@ function testWebTestRunner() {
 
+    // This causes the the tenth of the ten passes asserted in
+    // confirmStubResults().

the the.

But fixed this on commit.

Committed/pushed to 8.x, thanks!

Sorry this conflicted with #2016629: Refactor bootstrap to better utilize the kernel which was tagged 'avoid commit conflicts' and has been in and out of RTBC for a while, rolled back for now. Will need a quick re-roll.

That was an easy merge. As can be seen from the conflict file, it was just that the drupal_settings_initialize() that was in the diff context was removed by #2016629: Refactor bootstrap to better utilize the kernel. Here we go.

Found #2280501: Ensure web tests are encapsulated while trying to fix the tests here. :-(

Since #2016629: Refactor bootstrap to better utilize the kernel was reverted and needs to be reworked anyway (and the merge for them won't be very hard either), marking #59 RTBC again.

Reverted the revert, thanks!