[meta] Improve WebTestCase / BrowserTestBase performance by 50%

Enabled the caching by default for this patch and disabled MyISAM conversion for a first test.

And with MyISAM just for testing.

Uhm, Drupal CI applies https://www.drupal.org/files/issues/2551981-21-add-directory-option-to-r..., so this won't work. I am the 0.001% that runs into that issue.

So without nice arguments for now.

And again with MyISAM enabled.

Okay, I forgot to update the file paths after retrieving from cache ...

The test fails are a core bug, where the schema is wrongly installed (0) during installation of more than one module due to a static cache not being cleared properly.

Last test run showed from:

~ 10 min down to 6 min

Locally I see test times of 7 seconds instead of 28 seconds.

And test again with MyISAM

Edit: Down to 4 min 40 seconds!

Locally one test completes in 3 seconds! (with aggressive cache: 1 sec)

Test advanced cache (with more variations).

Lets try that once more a little less aggressively.

That worked, but the aggressive cache was way slower (16 min) - due to the many table copying.

One last check for a MySQL only optimization and then forward to Drupal 8.

@todo Need to check how to do this database agnostic anyway ...

And to 8.2.x

A first try, locally I only get 43 -> 32 s speed up, but lets see how many errors we get.

Now not _all_ using the same DB prefix and config dir ... uhm ...

There is a rootUser that needs to have the password known by the testBase. Also better method to write the settings that is not incompatible with UpdateTestBase.

#31: That is too bad, results show that the testing profile was efficient (28 min vs. 30 / 35 min), but that now with the cache in place, the opposite problem exists:

Because testing is so minimal, the speed-up is not as much as expected.

I have the three test fails fixed locally and attached is the interdiff.

For one test the caching just needs to be disabled.

I will try to run tests with a bloated testing profile next, but un-assigning for now.

My main goal here was to decrease my own local testing speed for faster development and that has been kinda-reached and I can always cache aggressively including modules for local dev.

+++ b/includes/module.inc
@@ -460,6 +460,7 @@ function module_enable($module_list, $enable_dependencies = TRUE) {
+      drupal_static_reset('drupal_get_schema_versions');

This is a core bug.

--

Short comment to the D7 version. The above is a legitimate core bug that I still need to file separately.

I am just leaving three screenshots here for motivation for the Drupal\comment\Tests\CommentPreviewTest, which uses standard profile. The XHProf is just for the setUp() of the test.

The following includes a lot of other optimization issues I have been testing out and is just a sneak preview:

The absolute minimum that I was able to achieve so far, without changing cache backends:

10 seconds (!) - This uses the aggressive cache of caching the whole setUp() and module installation().

This does not use aggressive caching and this does only cache the standard profile once (!) - 18 seconds

This last one does not do any installation caching at all, but optimizes e.g. the Config FileStorage and Container re-builds.

Memory might be an issue here however.

#36: Thank you for the review:

1. Indeed the functionality most likely will live in a trait as there is lots of duplicate code still.

2. Yes, this needs isCacheable() / setCacheable() functions.

3. Those globals pre-exist for config overrides, this just accesses them, but needs a comment - yes.

4/5. Yes, needs to be fixed. Agree.

6. This is pre-existing code how to get the ID - should probably add a helper.

7. Thanks!

8. Agree, and yes PHP does not have that. This function as copied from php.net help pages ... Any candidates known in Symfony or elesewhere? If not could live in Util ...

9. Thanks!

10. Will be completely removed, not sustainable (was from backdrop backport).

11. Thanks.

--

Yes, this will also cache the public files directory, but as we only cache the setup, we always have the same state.

I am making this a meta issue (which will get complete patches), so that child issues can be created for relevant milestones.

By analyzing the D8 test performance I found that the D7 approach of #2759197: [D7] Improve WebTestCase performance by 50% is not as impactful on Drupal 8, because of the minimal profile.

An analysis revealed the following important things:

- a) Container compilation is slow - mainly the Symfony steps [NEEDS ISSUE CREATED] - likely the biggest bang for the buck in Drupal 8 right now.

- b) BrowserTestBase needs a port now that WebTestBase is de-facto deprecated, likely the best approach is either a trait that shares common functionality or a Cacher class, which might allow D8 and D7 to share code more easily. Needs to be checked.

- c) Many tests do additional setup, which can be cached, so make it easy to cache for complex setUp(),s the whole step

- d) Not using minimal is partially slower, because DB copying is still slow, however it can be heavily parallelized => explore making DB copy from the snapshot faster by using mysqli directly and async queries.

- e) There is still some heavy writes happening after installation during module installation, e.g. the chained fast backend, cache tags invalidation and "state" setting. Consider moving temporarily to memory. (in general using db_merge() is slow in Drupal (as its delete and create always) - KeyValueStore could profit from using upsert's or try / catch instead.

That is MySQL only, but around 98% of our test time spent is on MySQL anyway.

Child issues from today (to be created):

- Use FileCache for infoParserDynamic.php, 5 loc, but lots of CPU saved, because system_rebuild_module_data() was calling

- Disable fast chained backend during installation of modules in tests (saves a lot of DB writes, similar to !drupal_installation_attempted() in ChainedFastBackendFactory)

e.g. a potential solution would look like:

  \Drupal\Core\Cache\ChainedFastBackendFactory::$disableFastChained = TRUE;
  // install
  \Drupal\Core\Cache\ChainedFastBackendFactory::$disableFastChained = FALSE;
  // Reload container from dump to reset cache services (not rebuild!)

- Use a ChainedFileCacheBackend + a FileStorageFileCacheBackend to (atomically) create FileCache entries on disk that are shared by all test processes.

I use that locally already and it works great - though I have to fix the atomicity for it still.

- Use pre-hashed passwords during tests for user creation to avoid calling crypt() again and again (saves around 100-200 ms per user creation call)

- Idea: Consider to create a buffering / queued chained cache backend + buffering cache tags invalidator:

The idea is to buffer all writes and invalidations, serve requests for the cache from static memory.

Have a flush() method to flush the memory backend contents to the database backend at the end, which disables the special functionality and it acts like a normal DB backend afterwards.

- Idea: Play more with caching the full container based only on "container.modules" and the normal container cache key as input. See if it can pass tests and how much memory is used.

Found out that:

Merge queries now use the Select before Insert, fallback to Update mechanism! (at least for SQlite) :) - YEAH - that makes K/V store updates faster!

Created #2796105: Move similar methods in BrowserTestBase / WebTestBase to a trait; untangle installDrupal() after talking with larowlan and dawehner.

That will make the speed enhancements of --cache and --cache-modules available to both WebTestBase and BrowserTestBase.

#50 Yeah, I don't think switching to MyISAM is necessarily that good of an idea anymore. And yes especially with the UFT8-MB4 changes, this won't work at all anymore - I think.

I think in the D7 issue is a patch, which does not have the MyISAM change:

#2759197-40: [D7] Improve WebTestCase performance by 50% is what I use personally.