Problem/Motivation

Previously, simpletest was using the main settings.php and all sorts of overrides were possible. Now this is gone.

I want to add mongodb to practically every test without needing to extend all of them to add it. Both DrupalKernel and CMI has overrides but the override is in settings.php and that is now set in stone.

Also, I suspect any site behind a reverse proxy needs $base_url to be set.

Proposed resolution

Check whether settings.testing.php exists in sites/default and if it does then copy and use it. Right now I have

$settings['drupal_bootstrap_config_storage'] = function () {
  drupal_classloader_register('mongodb', 'modules/mongodb');
  return Drupal\mongodb\Config\BootstrapConfigStorageFactory::get();
};
$config['system.module']['enabled']['mongodb'] = 0;

in sites.testing.php and this works well to add in MongoDB (and more modules besides). It'd be desirable to make DrupalKernel allow adding namespaces to avoid the closure but that's a different problem.

I added a little feature here for the benefit of settings.testing.php : a $test_class variable tells it which test it runs.

CommentFileSizeAuthor
#63 2229011-63-test-overrides.patch8.27 KBtstoeckler
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,751 pass(es), 2 fail(s), and 0 exception(s). View
#63 2229011-conflict.txt2.46 KBtstoeckler
#59 2229011-59-test-overrides.patch8.25 KBtstoeckler
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,740 pass(es). View
#59 2229011-57-59-interdiff-2.txt1.6 KBtstoeckler
#59 2229011-57-59-interdiff.txt1.12 KBtstoeckler
#57 2229011-57-test-overrides.patch8.44 KBtstoeckler
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,607 pass(es). View
#52 2229011-51-test-overrides.patch8.46 KBtstoeckler
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Unable to apply patch 2229011-51-test-overrides.patch. Unable to apply patch. See the log in the details link for more information. View
#52 2229011-47-52-interdiff.txt4.7 KBtstoeckler
#47 2229011-47-test-overrides.patch5.99 KBtstoeckler
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,267 pass(es). View
#47 2229011-45-47-interdiff.txt5.34 KBtstoeckler
#45 2229011_45.patch4.03 KBchx
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 70,442 pass(es). View
#32 2229011_32.patch2.08 KBchx
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 67,935 pass(es). View
#26 2229011_25.patch2.62 KBchx
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,987 pass(es). View
#25 2229011_25.patch4.88 KBchx
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,966 pass(es), 9 fail(s), and 0 exception(s). View
#18 2229011_14.patch2.22 KBchx
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,917 pass(es). View
#12 2229011_12.patch2.18 KBchx
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,916 pass(es). View
#10 2229011_10.patch2.41 KBchx
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Unable to apply patch 2229011_10.patch. Unable to apply patch. See the log in the details link for more information. View
#9 2229011_5.patch1001 byteschx
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,969 pass(es). View
#3 2229011_3.patch1000 byteschx
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,915 pass(es). View
copy_settings_php.patch884 byteschx
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 16,531 pass(es), 763 fail(s), and 761 exception(s). View
Members fund testing for the Drupal project. Drupal Association Learn more

Comments

Status: Needs review » Needs work

The last submitted patch, copy_settings_php.patch, failed testing.

sun’s picture

Could you clarify what you're trying to do and why you think this is needed, please?

The whole point of #2171683: Remove all Simpletest overrides and rely on native multi-site functionality instead was to remove each and every possible thing that could possibly leak from the test runner into tests.

All test results are muddied and inherently unreliable and invalid, if the same test does not produce the exact same result in all environments.

By including a custom settings.php instead of the empty default.settings.php, that is no longer guaranteed. Instead, there's a good chance for tests to fail out of nowhere on my machine, but "strangely" not on anyone else's machine. — Cause: settings.php.

chx’s picture

Issue summary: View changes
Status: Needs work » Needs review
FileSize
1000 bytes
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,915 pass(es). View

I want to add more modules to the tests without needing to extend every single test for this purpose. I would even be OK with having to write a special settings.php for this purpose which the attached patch does. It restores the regression while guaranteed to have no ill affects.

sun’s picture

I want to add more modules to the tests without needing to extend every single test

I don't really understand what this means?

  1. How can you add modules via settings.php?
  2. Why don't you use a base test class like all other tests in core that don't want to repeat themselves? That's why we introduced public static $modules...?
chx’s picture

Um. mongodb_aggregator replaces the aggregator storage, mongodb_node the node storage and so on. Now, I have two choices: a) write an empty class for every aggregator test in core which just contains a $modules = array('mongodb_aggregator') and repeat this for every. single. entity. b) override bootstrap config storage.

chx’s picture

Also, in general, I am not 100% we grasp the entirety of possibilities that need to be overridden during testing to keep Drupal ticking on a certain machine. Perhaps there are ini_set commands to run? Perhaps some config override? Drupal is big. Completely removing any and all possibilities to override a testing site is not a good idea and is an unnecessary regression. The solution presented puts the onus of not introducing uncertainty to whoever runs the test.

Finally, why I am arguing to fix a regression? It's a regression, we can fix it, it doesn't hurt anything. I really, really hate being in the core queue now especially against sun. Can we just move on please?

chx’s picture

Title: Tests are no longer modifyable » Tests are no longer modifiable
tim.plunkett’s picture

+++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
@@ -808,6 +808,10 @@ protected function setUp() {
+    if (file_exists(DRUPAL_ROOT . '/sites/default/settngs.testing.php')) {

settings is misspelled.

chx’s picture

FileSize
1001 bytes
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,969 pass(es). View

Thanks!

chx’s picture

FileSize
2.41 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Unable to apply patch 2229011_10.patch. Unable to apply patch. See the log in the details link for more information. View

I have finally got around to test this and install, if it can, will remove the writable bits so we need to restore them.

Status: Needs review » Needs work

The last submitted patch, 10: 2229011_10.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
FileSize
2.18 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,916 pass(es). View
sun’s picture

Title: Tests are no longer modifiable » Allow to run all core tests against custom/alternative storage engines
Category: Bug report » Feature request

This is not a regression, since a feature like this was never supported in the past.

Just because something was possible doesn't imply that it was supported and intended to be used and work that way. I hope we can agree on that principle.


Architecturally, this feature is asking for a mechanism that is similar to our existing Database driver negotiation — i.e., instead of hard-coding a particular driver, the system uses an available driver.

However, that mechanism does not exist for various storage concepts in core yet. Instead, the application hard-codes a default storage; which is a database based implementation in almost all cases.

One possible mechanism that is not only used by Symfony but also common practice in other languages and applications, is the concept of allowing $_ENV to override application-level configuration.

@msonnabaum mentioned this possibility a few times already; most recently in #2226761: Change all default settings and config to fast/safe production values

Whether the concept of introducing $_ENV (pretty much as in Symfony) would work for this use-case mostly depends on how complex those overrides are — i.e., whether they are just some static declarations (simply overriding service container definitions), or whether they involve dynamic PHP code.

→ A concrete example of what such a custom settings file actually contains would help to understand the actual requirements.

Based on that, we can check how this can be implemented in the most sensible way.

chx’s picture

Title: Allow to run all core tests against custom/alternative storage engines » Tests are no longer modifiable
Category: Feature request » Bug report
    $drupal_bootstrap_config_storage = Settings::get('drupal_bootstrap_config_storage');
    if ($drupal_bootstrap_config_storage && is_callable($drupal_bootstrap_config_storage)) {
      return call_user_func($drupal_bootstrap_config_storage);

Once you have the bootstrap storage, you can do pretty much whatever you want including overriding the list of modules then using your own service provider to change anything I want. I have this code snippet in my settings.php:

$settings['drupal_bootstrap_config_storage'] = function () {
  drupal_classloader_register('mongodb', 'modules/mongodb');
  return Drupal\mongodb\Config\BootstrapConfigStorageFactory::get();
};

works well. Yes, DrupalKernel needs to be patched so that namespaces can be added via settings because without it the well intended container_service_providers configuration option in DrupalKernel is problematic to use but -- DrupalKernel has a lot of pre-boot options, thanks god, and the one described above happens work.

Also, I absolutely have no idea where you got it "doesn't imply that it was supported and intended to be used and work that way". Please check who wrote #1784312-49: Stop doing so much pre-kernel bootstrapping where BootstrapConfigStorageFactory got introduced and then please enlighten me further on my own intentions. Or refer to the code comment "// Add site specific or test service providers" for that matter.

sun’s picture

Thanks for your feedback.

  1. We are aware of the available bootstrap service container override options. They do not apply to tests.

  2. The test environment is isolated on purpose. We've been working very long and hard to fix all the bugs in which the test runner environment leaked into tests. We also worked hard to ensure that tests yield the same results in all environments. Due to that, what you're reporting is the opposite of a regression; the test environment is intentionally not affected by the test runner's environment.

  3. It is still not clear whether that custom config storage override snippet presents all of the intended/necessary overrides to enable this feature. For example, why is that limited to overriding the bootstrap config storage, but does not change the active storage? Also, I assume that some storage engines will require connection credentials in order for this to work?

  4. Perhaps it's a bit early before having seen the full set of overrides, but TBH, just the given example settings.php snippet in #14 alone rather looks like a dirty hack and not something we'd want to support officially. I like the general idea of this feature request, but we should think some more about how this can be implemented cleanly.

It's too early to comment on the patch, but FWIW, the storage has to be overridden before the installer is invoked, not afterwards (as in the latest patch). Otherwise, Drupal would be installed using the default storage engine(s), instead of the custom, environment-specific one(s). The installer itself is an integral part of tests and involves plenty of storage operations, so custom storage engine overrides need to be applied already.

chx’s picture

> We are aware of the available bootstrap service container override options. They do not apply to tests.

Wrong. There's no we. There's only you. Cos for example I believe they do. What "we" are you talking about?

> The test environment is isolated on purpose.

The patch provides a separate settings.php for testing purposes so the isolation is kept.

> It is still not clear whether that custom config storage override snippet presents all of the intended/necessary overrides to enable this feature.

I admit not being able to get my tests pass but I suspect that's a problem on my end -- one would think that if you override the service reading the module list, that's the end of the story.

> alone rather looks like a dirty hack

In my opinion it's a simple closure; made necessary by the fact there is no namespace addition in DrupalKernel which is a problem (separate perhaps), but easily avoided. It is tested with is_callable; a closure is a 100% legit callable.

sun’s picture

Sorry, I believe that my comments were phrased in a collaborative, constructive, and productive way, but yet, the communication here creates an environment and atmosphere that isn't fun and that I don't want to participate in.

Happy to re-review this patch once a proper solution has been found for the actual feature request (which makes sense to me, given the amount of swappable storage engines in D8).

For what it's worth, I also think the issue summary needs to be rewritten to explain the actual use-case and intention, which was only vaguely described in comments later on. Likewise, a complete concrete example of intended service overrides would certainly help everyone to figure out the right architectural solution.

Thanks for contributing to Drupal.

chx’s picture

Issue summary: View changes
FileSize
2.22 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,917 pass(es). View

I figured what I broke in my code and yes this override works.

kattekrab’s picture

chx’s picture

I already did.

penyaskito’s picture

Status: Needs review » Needs work
  • $test_class appending confused me. Do we *really* need this variable? I would suggest to remove it, or at least adding an inline comment explaining the addition.
  • The possibility of adding a settings.testing.php file should be documented somewhere, not sure where exactly.
chx’s picture

Status: Needs work » Needs review

I still would like to see more reviews before attending to those. However, I am not sure how $test_class affects you -- the settings.php it is in lives only during a particular test. Also, the varaible is only made available for settings.testing.php and does not effect the rest of Drupal.

I guess https://drupal.org/simpletest a page under this + change notice would be necessary, yes.

chx’s picture

https://drupal.org/node/2230005 change notice. Documented $test_class as well.

chx’s picture

Issue summary: View changes

https://drupal.org/node/2230005 change notice. Documented $test_class as well.

chx’s picture

FileSize
4.88 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,966 pass(es), 9 fail(s), and 0 exception(s). View

Now with more inline comments.

chx’s picture

FileSize
2.62 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 66,987 pass(es). View

Ops, wrong patch.

The last submitted patch, 25: 2229011_25.patch, failed testing.

tstoeckler’s picture

Status: Needs review » Reviewed & tested by the community

All right. Tried this out locally and it works beautifully.

Looked at the code again and with the added code comments I have nothing to complain about.

Like @penyaskito I thought this feature should be documented somewhere at first, but I really don't know where. I thought about adding an empty settings.testing.php to the repository and adding some documentation to that, but that would just be noise for 99% of people who don't need to care about this. So let's just do this, maybe we can find a better way to document this in the future.

I do really think we should do this, though. The issue summary lists a very good example for which you can use this. @chx wanting to swap storage controllers in all tests is a IMO 100% legit other use-case. I imagine people writing custom storage backends and also cache backends will probably want to do that in testing as well. @sun has a similar use-case in #2161591-53: Change default active config from file storage to DB storage. So yeah, let's just do this.

tstoeckler’s picture

Re-read some of @sun's comments above. Some notes.

- A concrete example of how such a settings.testing.php file could look like is given in the issue summary. I thought of the use-case of swapping a different cache backend, which you can actually do in settings.php as well, fairly easily.
- The problem of reduced reliability is valid, but I don't think it's a killer for this feature. I.e. if something fails on one machine but not on another it's very trivial to figure out whether settings.testing.php was the cause, as you just need to look whether that file exists.
- I agree that we need some less hacky mechanism to swap things. I.e. right now it's pretty hard to swap out the storage controller of an entity type from settings.php. That's why @chx had to swap the config storage. That's not tangential to this issue, though. If there is some easier swappability mechanism I still would like to have the possibility of swapping it just for tests.
- I don't really care whether this is a regression or not. Let's just do this?

In other words I don't really understand the fundamental reservations @sun has against this. I see "there is no use-case for this" and "this mitigates test isolation" as points, but a) I think there are some use-cases that have been explained here and b) the reduced isolation is very minimal and also can be wanted, in the same use-cases that have been mentioned.

sun’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
    @@ -808,6 +809,21 @@ protected function setUp() {
         install_drupal($parameters);
    +    if (file_exists(DRUPAL_ROOT . '/sites/default/settings.testing.php')) {
    

    Potential storage service overrides should still apply to the installer already, because the installer pretty much writes out the entire system.

    I.e., the earlier patches in here were correct, only the later ones started to ignore the installer.

  2. +      copy(DRUPAL_ROOT . '/sites/default/settings.testing.php', $directory . '/settings.testing.php');
    

    As mentioned before, I generally support the idea of this feature request. — However, what makes me uncomfortable with the proposed solution is that executing a custom PHP file with arbitrary contents yields more or less the exact situation that triggered this issue:

    Instead of supporting a particular feature through a well-defined and explicit mechanism, people can create "bug reports" and claim that some arbitrary custom override is supposed to work.

    Can we limit this to a test.services.yml file?

    That would at least minimize the scope to overrides for the service container only. In addition, it would also no longer execute random PHP code, but a static/declarative definition.

  3. This patch only touches WebTestBase, which is confusing.

    Shouldn't the identical mechanism also work for DrupalUnitTestBase?

    The vast majority of storage related tests are DUTB [or will become so] — web tests ought to be limited to testing UI operations only.

  4. +++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
    @@ -808,6 +809,21 @@ protected function setUp() {
    +      file_put_contents($directory . '/settings.php', "\n\$test_class = '" . get_class($this) ."';\n" . 'include DRUPAL_ROOT . \'/\' . $conf_path . \'/settings.testing.php\';' ."\n", FILE_APPEND);
    

    I do not understand the $test_class addition here. Seems unrelated?

chx’s picture

Status: Needs work » Needs review

> I.e., the earlier patches in here were correct, only the later ones started to ignore the installer.

The patch ordering didn't change a bit since #3 which was the second patch posted to this issue and the first to pass. What earlier/later patches are you talking about? I will try to see what I can do with the installer; however that's slightly tricky business: we need the testing setting file to be included after the normal one but when the installer runs there's no settings.php yet -- if i write one with the include statement in it then, I believe, later I need to move it to the end which isn't trivial.

> As mentioned before, I generally support the idea of this feature request.

This is not a feature request. This was possible before the sites patch and isn't now. That's called a regression.

> Can we limit this to a test.services.yml file?

No as you can't do any overrides from services? Nor DrupalKernel nor CMI nor base_url nor anything similarly useful and necessary?

> I do not understand the $test_class addition here. Seems unrelated?

Yes. This is in the issue summary: this IS an additional feature but since it's within a line of code that is necessary anyways, I thought it not to be a big deal, but then again I need to deal with you so I guess I need to fight tooth and nail for every single whitespace not to mention a whole new variable that is a) useful for debugging b) doesn't effect Drupal in any way or form as it doesn't leave the scope of the function including settings.

chx’s picture

FileSize
2.08 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 67,935 pass(es). View

Here's a pre-installer version. It's only slightly less useful -- I believe you can not override anything set up install.php like database and config directory -- but then again we probably don't want to do those anyways and the code is much simpler as the directory is writeable at this point so for this I thank sun.

chx’s picture

Re. DrupalUnitTestBase I think those are less of a problem...? Ie if you are testing Drupal\aggregator\FeedStorage via a unit test then there's hardly anything we want to override. It is indeed the integration tests doing full bootstraps and so on that we need to override. I do not think there were overrides before either as there were no bootstraps in the tests.

sun’s picture

Thanks!

Just to be clear (again), I do support to add this feature in general, as also outlined much more in-depth below.

But yes, pretty much in line with the entire rest of the D8 architecture, facilities like this really ought to be explicitly and officially supported in a well-defined way.

That is, because it doesn't really help anyone if exactly one person figured out how to hack the system in order to inject a custom storage engine instead of the default one(s) into tests, whereas (1) the maintainers themselves have no idea of what exactly those hacks entail, (2) the testing framework undergoes continuous changes and no one knows whether those might break some undefined use-cases, and (3) everyone else having the same use-case will have to figure out and apply the exact same or similar hacks via custom, unreliable PHP code that doesn't meet any actual feature specification on their own.

Just in terms of product/maintenance aspects alone, that is the very definition of horror. ;-)

Therefore, the intention always was and still is to seek for a "proper" solution that (1) meets and resolves a concrete use-case and (2) actually can be officially supported for realz.

Now, if you could avoid the inappropriate personal attacks (and whichever "fight" you assume to have with anyone) in your communication, then I'd be happy to continue to work with you to make this happen, because I think it's a good feature to have in case it's well-scoped and well-defined. - But to be clear, your communication here thus far does not fly; if you continue to do that, then I'm out of this issue.

  1. re: WebTestBase vs. DUTB:

    Here's where I'm coming from: Having a look at a typical class hierarchy of e.g. FieldUnitTestBase (DUTB) — all of those tests are entity/field storage related.

    My assumption is that it's pretty much exactly all of those storage related tests that you are actually interested in, in case you have the use-case of running all tests against a custom storage engine.

    The storage related services actually get injected into many other services (Properly Decoupled Spaghetti++), so in almost all cases, it is sufficient to replace the storage service with a different implementation, and all of the dependent tests will actually use it.

    As mentioned previously, web tests only exist to test for UI-level aspects; e.g., that clicking link Foo leads to page Bar, and pressing form button X yields Y... — Aside from a couple of very old legacy web tests that were not converted/separated yet, I'm not actually able to see the point of running web tests against a custom storage engine...?

    Don't get me wrong, I think it's fine to include web tests, but that would primarily happen for consistency only — the real meat of storage related tests (thankfully) are not really web tests anymore (in D8).

    This means config, entity, field, key/value, cache, etc.pp... it's completely pointless to use WebTestBase for storage related tests, and unless I'm terribly mistaken, the vast majority of these tests is using DrupalUnitTestBase in D8 already.

    DrupalUnitTestBase uses DrupalKernel, the consumption/registration of additional service providers happens within DrupalKernel, so additional service providers are consumed by DUTB, too.

  2. It's mainly the aforementioned considerations that lead me to the conclusion that we are actually seeking for a (officially supported) mechanism for replacing/overriding storage engines with custom ones in tests, in a well-defined and controlled way.

    I'd expect that to work with e.g. a test.services.yml service provider file in the site directory, which essentially replaces/overrides all the service definitions from core.services.yml of interest, so that they point to implementations of a custom storage engine instead; for example:

    services:
      keyvalue:
        class: Drupal\mongo\KeyValueStore\MongoDbKeyValueFactory
    

    I'm not sure why you're saying that the YAML service provider approach cannot be used for replacing service definitions (as that is one of its only two purposes), so if there is a bug with that (possibly the merging algorithm?), then we should fix that.

    An approach like this would make at least me much more comfortable with this feature addition. Because it means that we have a crystal-clear way for overriding default storage services in tests. By removing the randomness of completely custom executed PHP code, the feature can be officially supported (which is also in your interest).

    In fact, a well thought-out solution here will also allow Drupal 8 core itself to improve its test coverage through minimally customized PIFR environments. This happens to be an actively discussed topic currently, whereas arguments of others even go up to the extent of removing all code from core that is not fully covered by tests — whereas alternate implementations are only not tested, because there is no easy + officially supported way to inject them into the testing environment.

    For example, #2160433: [policy, no patch] Move PostgreSQL driver support into contrib

    That is pretty much the main reason for why I'm in support for the general, underlying idea of this feature request.

    And as a very concrete additional example, an officially supported feature like this would allow us to battle-test the entire core test suite against #2208617: Add key value entity storage (which apparently presents the canonical abstract implementation for custom storage engines)

  3. I just noticed that the patch hard-codes sites/default as the test-runner/parent site directory.

    sites/default is not guaranteed to be the site directory of the test-runner/parent-site.

    Perhaps (much) more interestingly, by consulting the proper parent site, the facility of applying custom storage service overrides can actually be tested through an automated test.

    I.e., a recursive web test like SimpleTestTest, but for the sake of core developer sanity, let's please create a separate test class for this new test... ;-) — With above mentioned support for DUTB, the test executed in the child site can actually just be a DUTB test (improving test performance).

    We do have quite a range of alternative implementations for various services in core already, so testing an actual custom service override should be possible. (Above mentioned k/v entity storage would make most sense from my perspective, since it actually maps 1:1 to the actual use-case, but yeah, that hasn't landed yet...)

    Test coverage == clearly defined expectations == supported feature

    (Just to be clear: Executing random PHP code does not meet the requirement of that equation.)

  4. re: $test_class:

    I've read the issue summary, but all it states is this:

    a $test_class variable tells it which test it runs.

    Doesn't explain how that is useful? The test class is always reported by the test runner already. Why does the system under test actively need to know the test class name?

chx’s picture

Status: Needs review » Closed (won't fix)

I will just add a core patch to the mongodb profile.

chx’s picture

To clarify: you are not interested in working together. You are interested in having your way and nothing else. You have always done this even after Dries himself called you out on it. I tried to put up a fight but I am sick of it.

Edit: if it's not clear, your repeated use of "official" (you do not get to define official) and "feature request" (it is a crystal clear regression) is what caused me to quit this issue. I could continue pointing out your FUD and outright lies as I did in the comments above which you constantly ignore but I won't any longer.

chx’s picture

Status: Closed (won't fix) » Needs review

Sigh. #34 is entirely off topic, a tried and true tactic to unhinge me. So far, sun suggested implementing the dev/prod switch for Symfony, making service overrides happen in yaml instead of the service provider mechanism that is in HEAD and allowing overrides in DUTB -- practically anything and everything as long as it is not my patch. Now, DUTB overrides were not possible before; perhaps it should be but not in this issue

To clarify: this issue is about restoring the capability that was lost with the move to multisite and nothing else. We can do more in followup issues.

> I just noticed that the patch hard-codes sites/default as the test-runner/parent site directory.

Nope, HEAD does that:

     copy(DRUPAL_ROOT . '/sites/default/default.settings.php', DRUPAL_ROOT . '/' . $this->siteDirectory . '/settings.php');

my patch merely factors out DRUPAL_ROOT . '/' . $this->siteDirectory into a variable. If you mean the settings.testing.php being copied from this dir, if default.settings.php can be, so can be the testing file, I guess.

tstoeckler’s picture

OK, I'll take another stab at providing reason why I think the current approach is feasible and correct.

Mind you that I thought long and hard about this and re-read each of @sun's comments multiple times. As @sun's mentee I tend to agree on a lot of things with him and I feel that often when he shares architectural concerns with an otherwise agreed upon patch it's for the better. In this issue, I cannot come around disagreeing, though.

WebTestBase vs. DrupalUnitTestBase

As DrupalUnitTestBase does not do real unit tests and - as @sun points out - a lot of DrupalUnitTestBase-tests do storage-related things they would lend themselves extremely well for the swappability that this issue wants to introduce.

So I think we all agree that in the end we want to provide a way of overriding / swapping things for both WebTestBase and DrupalUnitTestBase.

However, the question on whether to focus this issue primarily on WebTestBase or on DrupalUnitTestBase is tied to the general architectural decision of whether to use a settings.testing.php file or not in a chicken-and-egg manner:

- If you - like @sun does - have reservations against the settings.testing.php file anyway and think that the patch needs to be re-architected anyway it seems natural to introduce the re-architected code in TestBase, so that both DrupalUnitTestBase and WebTestBase can utilize it, solving the duplicity problem nicely and elegantly.

If - like @chx and I do - believe, however, that the settings.testing.php approach is correct then it makes sense to introduce this behavior for WebTestBase only for now because that is the only type of test that currently has a settings.php file in the first place. We could then - in a follow-up - explore creating a settings.php file for DrupalUnitTestBase-tests as well. This would make sense regardless of this issue (... in my opinion at least. Let's not bikeshed this detail here, though, please.)

So the question on whether or not DrupalUnitTestBase should receive a similar treatment to WebTestBase should be postponed on resolving the other, larger question. Let's please just leave it out entirely for now.

settings.testing.php vs. explicit service overrides

In theory explicit overrides are much preferable over including a PHP file which can run arbitrary code, I think there is no question about that. The question is rather how feasible it is that we can provide an explicit mechanism for overriding all of the things that we can override.

Service overrides
@sun discussed service overrides above, so let's look at those in detail first. @sun mentions the possibility of simply specifying overrides in a YAML file. Looking at DrupalKernel that should actually work, but I'm pretty sure that literally noone is aware of that "feature". It is in the same way "not official" - i.e. not documented, not explicitly supported, not tested, etc. - as the settings.php overrides for tests are claimed to have been "not official". Proof of this is (among other things) LanguageServiceProvider. Specifically the overriding of the language manager, which could in theory (given that that actually works) be // Add site specific or test service providers.
easily done in language.services.yml. This override is often used as the canonical example of explaining to Drupal 7 developers why the whole "service" concept makes sense. Thus, I can say with certainty that it is using the "official" way of overriding a service. Therefore to not convolute this issue even further this point must be explored in a separate issue. Only when it is proven that this actually works, a testing-specific services.yml can be discussed. For now we must assume that a separate ServiceProvider class is needed for that. Note that this is only a minor point, because we need some sort of discovery mechanism (see below) either way: either for the service provider or for the YAML file.

In theory it would not be a problem to allow to put a TestingServiceProvider class in /sites/simpletest which gets picked up when building the test container. In theory for two reasons:

  1. If implementing this correctly, we would have to create a separate TestingKernel which would pick this class up. I don't think adding a drupal_valid_test_ua() check to DrupalKernel is going to fly (of course I don't get to decide that, but I sincerely hope that that wouldn't fly!). That however would require to establish an entire separate bootstrap process for tests. That might become possible with #2016629: Refactor bootstrap to better utilize the kernel but TBH a pipe-dream at this point.
  2. We already have a facility to allow to:
    // Add site specific or test service providers.

    (DrupalKernel, l. 238) but that requires setting global $conf (yes, it's still $conf...) somewhere. Where is global $conf intended to be set? In settings.php, of course. When reading that particular part of DrupalKernel the current approach of a test-specific settings.php feels quite natural.

    Note: Let's please not discuss the fact whether this issue is a regression or not any more. This issue is already very loaded without that detail. I realize that the quoted code comment could be seen as an argument for one side in that fight. But again, please let's ignore that detail.

In conclusion:
There could be cleaner ways of providing one-off service providers, for a custom site or for tests. And hopefully we will get there to enable that eventually. But as it stands, the one that is used in core is by setting a global in a settings.php file.

Settings overrides
The example provided by @chx uses a Settings override to specify the bootstrap config storage, in order to override the module list. Although that is very hacky, it is a working (I haven't tried it, but I believe @chx if he says it works) and probably still the cleanest way of achieving that. In general I believe it makes sense to allow for settings overrides. But at the very least until someone provides a different way of achieving the same result I think we must support this.

So how do we override Settings? With a settings.php file, of course. It's really just that. I've heard @dawehner talk about moving Settings declaration into YAML, but - again - that's just a pipe dream at this point. And when you include a PHP file, there is no middleground: either you include it or you don't, but if you include it you can't prevent people doing all sorts of crazy things in there. I.e. there is no way to have a (PHP) file that's just for overriding settings.

Override of globals
@chx mentions the potential need to override $base_url for tests. I can't claim that I fully understand that use-case but it hasn't been refuted so far, so I can only assume it's a valid use-case that needs to be supported. This is - again - a situation where we could certainly do better. I.e. we shouldn't be relying on globals at all at this point. But the fact of the matter ist that we do, and - for now - we need to support that. Here the argument is the same as for the settings overrides. Globals are overriden in a PHP file that is included (== 'settings.php'), end of story.

The conclusion is similar to the one for service overrides already: We have a system for overriding various parts of the system in a one-off fashion. That system is settings.php. It could be improved and replaced in parts or entirely to make for a much cleaner system. And everyone would applaud that, I think. But currently that is the system that we have. So now that we have a use-case for overriding various parts of Drupal in a one-off fashion - namely for tests - we should use that system. KISS.

Test isolation

Surely the settings.testing.php would by its nature reduce isolation and reliability of tests. @sun, you mention "developer sanity" being at stake and that debugging would be greatly impaired by this change. If thought about this for a long time, but I could not come up with a single reason why that would be the case. To maybe make it easier to find the specific point of disagreement, I will state some things that I would consider consentual, but in aggregate they would dispute your claim. Therefore I assume that you will disagree with at least one of those points:

  1. Many developers will not ever see a settings.testing.php file: It is primarily purposed for testers (i.e. writers) of alternate storage backends / DB drivers / cache backends / ... Neither in core nor in contrib more than a few percent of people will use this.
  2. A settings.testing.php file will generally be single-purpose: It will be used to harcode-enable the MongoDB modules in tests or to store config with the key-value store (or whatever) but not both. While that's absolutely possible it doesn't make much sense to include various source of failures into one environment.
  3. If you do encounter a test failure that someone else cannot reproduce it will take literally about 10 seconds (at most) to find out whether you have a settings.testing.php file on your system. If you do have one, just (temporarily) remove it, and re-run the test. If your test now passes you have the reason for your failure. If not, there must be another issue. In other words: Specifically the issue of impaired debugging I fail to see conmpletely.

Misc

Let's please leave the following items out of the discussion completely. They are implementation details and can be discussed once we have agreed on a general solution.

  • Whether or not to provide a $test_class in settings.testing.php
  • Whether to take the settings.testing.php from /sites/default or from $conf_path

Let's please leave the discussion about this completely out of this for the sake of not loading this issue even more. I'm pretty sure that @chx would be happy to remove that bit of code if there were no other objections.

chx’s picture

Thanks for the exhaustive summary. I will only add this:

> @chx mentions the potential need to override $base_url for tests. I can't claim that I fully understand that use-case

You have N webfrontends behind a reverse proxy and you want to test the whole setup. Say, your site is example.com and the frontends are web1.example.com , web2.example.com etc. What you want is $base_url = 'example.com';. Of course, you will also want to test individual servers but especially with session tests it is sensible to try to emulate an outside visitor.

dawehner’s picture

TL;DR
We need something to improve our code quality by enhancing our test dimensions, though more documentation on both the new settings.testing.php
and existing override mechanism should get some focus.

In an ideal world we would not only have proper abstractions and layers
on the code side but we would have similar concepts on the tests.

Pure unit tests aka. UnitTestBase
For them you don't have container or whatever, so go along there is nothing to see here.
Code integration tests aka. DrupalUnitTestBase
These tests just rely on the DB + enviroment (kernel etc.) so you want to ensure
that your alternative storage backend still works on the pure API side. Therefore
you most of the time "just" need a way to swap out single services. YML files seems
to be totally okay for that. These tests most of the time don't deal with a
request object so for example changing the base url is not really needed. For everything else using the global is currently deprecated anyway but you would use a different request context service which does some fancy things. This should be entirely possible using yml files/container passes.

One problem could be that the DUTB tests rely on storage details, which aren't part
of the public API. Being able to swap out the underlying storage would help us to
cover those problems.

Webtests
These tests rely on a fully functional drupal so much like people do have a settings.php to change stuff on
their site we need to have test coverage for such chanages. Having a settings.test.php seems to be a pragmatic approach for that. I think we all agree that we should move away from the globals, try to avoid settings as they are a sign of work before bootstrap etc.
I really wonder whether we want to provide some default testing settings php which explains HOW you override
specific things in the right way.

On top of that I am really looking forward to being able to use drupal with composer which would allow you
to have potentialy control over index.php and a maybe site specific drupal kernel at the end of the day.

+++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
@@ -799,6 +800,13 @@ protected function setUp() {
+    if (file_exists(DRUPAL_ROOT . '/sites/default/settings.testing.php')) {

Just in general: If you do create this file you will be aware of the problems you might have, so if stuff fails you will know that you might first run core only tests first.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

@chx promised to improve the documentation in the future, even I would like to see something in Drupal itself, but well you can't have just the cherries.

chx’s picture

Well, we have a change notice already and I promised to add the info to the simpletest handbook -- I can also write a blog post. I do not know where else would I doc this.

sun’s picture

+++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
@@ -775,7 +775,8 @@ protected function setUp() {
+    copy(DRUPAL_ROOT . '/sites/default/default.settings.php', $directory . '/settings.php');

@@ -799,6 +800,13 @@ protected function setUp() {
+    if (file_exists(DRUPAL_ROOT . '/sites/default/settings.testing.php')) {

The difference between these two is that there is only one default.settings.php file in Drupal and that happens to be located in /sites/default/default.settings.php.

The installer always copies the files from that location. There is no other source.

Hence the "off-topic" reference to #1757536: Move settings.php to /settings directory, fold sites.php into settings.php, which (finally) moves this file into /core/default.settings.php.

Tests are not necessarily executed from within the default site.

Therefore, it is wrong to check the default site directory instead of the actual site directory of the test runner.


Test coverage?

webchick’s picture

Status: Reviewed & tested by the community » Needs review

It seems like this is still under discussion.

chx’s picture

FileSize
4.03 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 70,442 pass(es). View

Moved to site directory, added test. Sorry, no interdiff, the interdiff is larger than the patch. It's quite easy to test this as the simpletest self testing framework is in place , just add the file and two asserts it is copied and included.

tstoeckler’s picture

Status: Needs review » Needs work

This will need an update after #2241633: Simplify site-specific service overrides. I.e. we will want to allow a test-specific testing.services.yml as well.

Will try to work on this this week.

tstoeckler’s picture

Status: Needs work » Needs review
FileSize
5.34 KB
5.99 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,267 pass(es). View

Here we go. Added support for a testing.services.yml*. And also added test coverage for that.

I also slightly expanded the existing test coverage to also test the $test_class local variable. I verified that removing that actually results in fails in SimpleTestTest.

* Even though we have a settings.testing.php - which is consistent with settings.local.php - I called this testing.services.yml to be consistent with core.services.yml. This has the result that we have settings.testing.php and testing.services.yml right next to each other which seems inconsistent, but anything different would be (even more) mind-boggling to my mind.

chx’s picture

This looks good , the if ($test_class)

dawehner’s picture

  1. +++ b/core/modules/simpletest/lib/Drupal/simpletest/TestBase.php
    @@ -54,6 +54,13 @@
    +   * The site directory of the original parent site.
    +   *
    +   * @var string
    +   */
    +  protected $originalSite;
    +
    

    was that previously just not documented? I don't see whether this is set.

  2. +++ b/core/modules/simpletest/lib/Drupal/simpletest/Tests/SimpleTestTest.php
    @@ -237,7 +272,7 @@ function confirmStubTestResults() {
     
    -    $this->assertEqual('6 passes, 5 fails, 2 exceptions, 1 debug message', $this->childTestResults['summary']);
    +    $this->assertEqual('10 passes, 5 fails, 2 exceptions, 1 debug message', $this->childTestResults['summary']);
    

    Can we put something like a @see to the part of the code which produces the actual assertions? stubTest()

  3. +++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.php
    @@ -843,6 +844,21 @@ protected function setUp() {
     
    +    // Allow for test-specific overrides.
    +    $settings_testing_file = DRUPAL_ROOT . '/' . $this->originalSite . '/settings.testing.php';
    +    if (file_exists($settings_testing_file)) {
    +      // Copy the testing-specific overrides in place.
    +      copy($settings_testing_file, $directory . '/settings.testing.php');
    +      // Add the name of the testing class to settings.php and include the
    +      // testing specific overrides
    +      file_put_contents($directory . '/settings.php', "\n\$test_class = '" . get_class($this) ."';\n" . 'include DRUPAL_ROOT . \'/\' . $conf_path . \'/settings.testing.php\';' ."\n", FILE_APPEND);
    +    }
    +    $settings_services_file = DRUPAL_ROOT . '/' . $this->originalSite . '/testing.services.yml';
    +    if (file_exists($settings_services_file)) {
    +      // Copy the testing-specific service overrides in place.
    +      copy($settings_services_file, $directory . '/services.yml');
    +    }
    

    DUTB tests could also profit at least from the custom testing.services.yml file.

chx’s picture

core/modules/simpletest/lib/Drupal/simpletest/TestBase.php
1013:    $this->originalSite = conf_path();

DUTB is a separate issue still, at least I think so.

dawehner’s picture

@chx
Thank you for the pointer. I don't have tools available to look it up at the moment.

Well, tobias agreed that it makes sense to support DUTB as well.

tstoeckler’s picture

FileSize
4.7 KB
8.46 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Unable to apply patch 2229011-51-test-overrides.patch. Unable to apply patch. See the log in the details link for more information. View

OK, here we go. Added a bunch of inline comments to SimpleTestTest. Also added the same hunk of code to DrupalUnitTestBase.

I wanted to write a test for that, but we don't have a sort of SimpleTestTest-Inception-Like Test for DUTB, so that's not really possible. I did verify that the overrides do work for DUTB tests, however.

@dawehner: RTBC? :-)

tim.plunkett’s picture

tstoeckler’s picture

Status: Needs review » Needs work

The last submitted patch, 52: 2229011-51-test-overrides.patch, failed testing.

tstoeckler’s picture

Status: Needs work » Needs review
FileSize
8.44 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,607 pass(es). View
dawehner’s picture

Status: Needs review » Needs work

Thank you for adding the explanations!

+++ b/core/modules/simpletest/lib/Drupal/simpletest/Tests/SimpleTestTest.php
@@ -184,23 +216,41 @@ function testWebTestRunner() {
+    // This causes

... what ?

tstoeckler’s picture

Status: Needs work » Needs review
FileSize
1.12 KB
1.6 KB
8.25 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,740 pass(es). View

Here we go. This fixes the test comments.

Two interdiffs because I merged 8.x in between. Also, if you - like me - fail at math: "5th to 9th" is actually 5 things, not 4...

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

Whether this is a problem of people or of the english language itself is not clear for me.

catch’s picture

Status: Reviewed & tested by the community » Fixed
+++ b/core/modules/simpletest/src/Tests/SimpleTestTest.php
@@ -184,23 +216,42 @@ function testWebTestRunner() {
 
+    // This causes the the tenth of the ten passes asserted in
+    // confirmStubResults().

the the.

But fixed this on commit.

Committed/pushed to 8.x, thanks!

catch’s picture

Status: Fixed » Needs work

Sorry this conflicted with #2016629: Refactor bootstrap to better utilize the kernel which was tagged 'avoid commit conflicts' and has been in and out of RTBC for a while, rolled back for now. Will need a quick re-roll.

tstoeckler’s picture

Status: Needs work » Needs review
Related issues: +#2016629: Refactor bootstrap to better utilize the kernel
FileSize
2.46 KB
8.27 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,751 pass(es), 2 fail(s), and 0 exception(s). View

That was an easy merge. As can be seen from the conflict file, it was just that the drupal_settings_initialize() that was in the diff context was removed by #2016629: Refactor bootstrap to better utilize the kernel. Here we go.

Status: Needs review » Needs work

The last submitted patch, 63: 2229011-63-test-overrides.patch, failed testing.

tstoeckler’s picture

Found #2280501: Ensure web tests are encapsulated while trying to fix the tests here. :-(

tstoeckler’s picture

Status: Needs work » Reviewed & tested by the community

Since #2016629: Refactor bootstrap to better utilize the kernel was reverted and needs to be reworked anyway (and the merge for them won't be very hard either), marking #59 RTBC again.

catch’s picture

Status: Reviewed & tested by the community » Fixed

Reverted the revert, thanks!

  • Commit 78d18c8 on 8.x by catch:
    Issue #2229011 by chx, tstoeckler: Fixed Tests are no longer modifiable.
    

  • Commit 2cd0b09 on 8.x by catch:
    Revert "Issue #2229011 by chx, tstoeckler: Fixed Tests are no longer...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.