Cache test starting point to increase testing speed by about 25%

With D7 sites, I use the module Simpletest turbo which caches the result of tests' setUp() function so the starting point of a test can be carried over from one test to another without going through all the steps to install Drupal.

To see if such an approach could benefit Drupal 8, I started by installing Drupal 8.0-alpha12 with a MySQL database, and running the four first tests in the Node group, from the GUI.

On my computer (Mac OS X with MAMP), these tests were performed in 1 minute 9 seconds during the first run. (I ran the same tests twice, and both times the time taken was the same.) The time it takes to run a test can be broken down as:

  • 69 seconds total
  • 24 seconds (35%) for install_drupal (4 * 6 seconds).

If we could apply the principle of Simpletest turbo, I think it not unreasonable to shave 2/3 off the 24 seconds it takes to install Drupal by reusing a previous installation database rather than reinstalling everything. If this can be achieved, the four tests in our sample would take 69-(24*.66), or 53 seconds. By extrapolating, we could make the testbot 25% more efficient overall.

CommentFileSizeAuthor
#11 2283553-11-core-8.x-simpletest-cache-setup-DUMMY-PATCH-FOR-COMPARISON.patch535 bytesalberto56
#6 2283553-6-core-8.x-simpletest-cache-setup-DUMMY-PATCH-FOR-COMPARISON.patch418 bytesalberto56
#6 2283553-6-core-8.x-simpletest-cache-setup.patch5.51 KBalberto56
#3 Screen Shot 2014-06-10 at 11.18.06 AM.png223.78 KBalberto56
#2 2283553-2-core-8.x-simpletest-cache-setup.patch4.64 KBalberto56

Comments

alberto56’s picture

alberto56 commented
Issue summary: View changes
alberto56’s picture

alberto56 commented
Status: Active » Needs review
StatusFileSize
new 2283553-2-core-8.x-simpletest-cache-setup.patch4.64 KB

This patch is a proof of concept and has only been tried with a MySQL database on the GUI (not the commande line), but I managed to get a 14.7% performance increase using Simpletest.

To use the patch:

  • Download the latest version of D8
  • Enable Testing (simpletest)
  • Go to admin/config/development/testing
  • Unfold the Node test group and select the first eight (8) tests, from "Load multiple nodes" to "Node access on any table", inclusively.
  • Run the test in the GUI without the patch. On my machine I am getting 3 min. 58 seconds (see image in comment #3)
  • Now apply the patch
  • Go back to admin/config/development/testing
  • Run the same eight (8) tests you did earlier.
  • On my machine I am getting 3 min. 23 seconds, a 14.7% speed increase if my math is correct (see image in comment #3).

Note: this is only a proof of concept. The following could be done:

  • Try with Sqlite
  • Try on the command line
  • For now if you load the site on another window the test in the first window will fail
  • For now the test logs you out of your host environment sometimes.
  • more optimization, perhaps: anything other than the database creation done during initial installation could be cached as well.
  • Allow subclasses' setUp() function to benefit from this as well: if a core, contrib or custom test class uses a setUp() function which takes very long, it should be possible for the developer to cache that as well. When I apply the concept of caching the installation tables to a site with a complex site deployment module on Drupal 7, my speed increase is close to 75%.
alberto56’s picture

alberto56 commented
StatusFileSize
new Screen Shot 2014-06-10 at 11.18.06 AM.png223.78 KB

Here is the screenshot

Status: Needs review » Needs work

The last submitted patch, 2: 2283553-2-core-8.x-simpletest-cache-setup.patch, failed testing.

alberto56’s picture

alberto56 commented
Title: Cache test starting point to increase testing speed by about 25% » Cache test starting point to increase testing speed by about 15%

OK, maybe 25% is exaggerated. 15% is still respectable. Updating title.

alberto56’s picture

alberto56 commented
StatusFileSize
new 2283553-6-core-8.x-simpletest-cache-setup.patch5.51 KB
new 2283553-6-core-8.x-simpletest-cache-setup-DUMMY-PATCH-FOR-COMPARISON.patch418 bytes

In an attempt to get some metrics on the command line, I ran the following command on my computer with PHP 5.5 and a MySQL database.

php ./core/scripts/run-tests.sh --url http://localhost/2283553-test-speed/ --class "Drupal\node\Tests\NodeAccessBaseTableTest","Drupal\node\Tests\NodeAccessFieldTest","Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest","Drupal\node\Tests\NodeAccessLanguageAwareTest","Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest"

The idea was to run a small subset of available tests to see the speed difference with this approach.

Note that the patch in #2 no longer applies to 8.x, so I modified it slightly and applied it. I also am now clearing tables before starting the command line operation.

Without the patch the test ran in 2 min 6 sec
With the enclosed patch the test ran in 1 min 23 sec

Which is 34.13% faster according to my calculations. It should be noted that the more tests are run, the bigger the speed increase will be (in theory anyway). So perhaps the 15% in the title is a bit conservative.

To have a better idea of the actual speed increase, I am including two patches
here, one which changes nothing in Drupal, and the other which implements the solution discussed herein. The idea then is to compare the time it takes the testbot to test each patch.

Here is my output:

php ./core/scripts/run-tests.sh --url http://localhost/2283553-test-speed/ --class "Drupal\node\Tests\NodeAccessBaseTableTest","Drupal\node\Tests\NodeAccessFieldTest","Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest","Drupal\node\Tests\NodeAccessLanguageAwareTest","Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest"

Drupal test run
---------------

Tests to be run:
  - Drupal\node\Tests\NodeAccessBaseTableTest
  - Drupal\node\Tests\NodeAccessFieldTest
  - Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest
  - Drupal\node\Tests\NodeAccessLanguageAwareTest
  - Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest

Test run started:
  Saturday, August 9, 2014 - 23:06

Test summary
------------

Drupal\node\Tests\NodeAccessBaseTableTest                    159 passes                                      
Drupal\node\Tests\NodeAccessFieldTest                         50 passes                                      
Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest     142 passes                                      
Drupal\node\Tests\NodeAccessLanguageAwareTest                 91 passes                                      
Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest           17 passes                                      

Test run duration: 2 min 6 sec

$ patch -p1 < 2283553-6-core-8.x-simpletest-cache-setup.patch 
patching file core/modules/simpletest/simpletest.module
patching file core/modules/simpletest/src/WebTestBase.php
patching file core/scripts/run-tests.sh
$ php ./core/scripts/run-tests.sh --url http://localhost/2283553-test-speed/ --class "Drupal\node\Tests\NodeAccessBaseTableTest","Drupal\node\Tests\NodeAccessFieldTest","Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest","Drupal\node\Tests\NodeAccessLanguageAwareTest","Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest"

Drupal test run
---------------

Tests to be run:
  - Drupal\node\Tests\NodeAccessBaseTableTest
  - Drupal\node\Tests\NodeAccessFieldTest
  - Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest
  - Drupal\node\Tests\NodeAccessLanguageAwareTest
  - Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest

Test run started:
  Saturday, August 9, 2014 - 23:08

Test summary
------------

Drupal\node\Tests\NodeAccessBaseTableTest                    159 passes                                      
Drupal\node\Tests\NodeAccessFieldTest                         50 passes                                      
Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest     142 passes                                      
Drupal\node\Tests\NodeAccessLanguageAwareTest                 91 passes                                      
Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest           17 passes                                      

Test run duration: 1 min 23 sec
alberto56’s picture

alberto56 commented
Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 6: 2283553-6-core-8.x-simpletest-cache-setup-DUMMY-PATCH-FOR-COMPARISON.patch, failed testing.

Status: Needs work » Needs review

alberto56 queued 6: 2283553-6-core-8.x-simpletest-cache-setup-DUMMY-PATCH-FOR-COMPARISON.patch for re-testing.

Status: Needs review » Needs work

The last submitted patch, 6: 2283553-6-core-8.x-simpletest-cache-setup-DUMMY-PATCH-FOR-COMPARISON.patch, failed testing.

alberto56’s picture

alberto56 commented
StatusFileSize
new 2283553-11-core-8.x-simpletest-cache-setup-DUMMY-PATCH-FOR-COMPARISON.patch535 bytes

Seems by dummy patch is not applying. Here is a new attempt. Please do not use this patch; use the one at #6, above. The one here is simply here to have a baseline on how much time the testbot normally takes to run a test (to be taken with a grain of salt given the various factors affecting performance).

alberto56’s picture

alberto56 commented
Status: Needs work » Needs review

The last submitted patch, 6: 2283553-6-core-8.x-simpletest-cache-setup.patch, failed testing.

alberto56’s picture

alberto56 commented
Status: Needs review » Needs work

https://qa.drupal.org/pifr/test/841008 ran in 33 min 35 sec
https://qa.drupal.org/pifr/test/841018 ran in 33 min 42 sec

A bit anticlimactic. But hope is not lost.

alberto56’s picture

alberto56 commented

The reason why the testbot doesn't show a speed increase is because the patch does not increase speed when concurrency is used:

php ./core/scripts/run-tests.sh --concurrency 8 --url http://localhost/2283553-test-speed/ --class "Drupal\node\Tests\NodeAccessBaseTableTest","Drupal\node\Tests\NodeAccessFieldTest","Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest","Drupal\node\Tests\NodeAccessLanguageAwareTest","Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest"

Drupal test run
---------------

Tests to be run:
  - Drupal\node\Tests\NodeAccessBaseTableTest
  - Drupal\node\Tests\NodeAccessFieldTest
  - Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest
  - Drupal\node\Tests\NodeAccessLanguageAwareTest
  - Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest

Test run started:
  Sunday, August 10, 2014 - 01:29

Test summary
------------

Drupal\node\Tests\NodeAccessLanguageAwareTest                 91 passes                                      
Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest     142 passes                                      
Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest           17 passes                                      
Drupal\node\Tests\NodeAccessFieldTest                         50 passes                                      
Drupal\node\Tests\NodeAccessBaseTableTest                    159 passes                                      

Test run duration: 1 min 19 sec

$ patch -p1 < 2283553-6-core-8.x-simpletest-cache-setup.patch 
patching file core/modules/simpletest/simpletest.module
patching file core/modules/simpletest/src/WebTestBase.php
patching file core/scripts/run-tests.sh
$  php ./core/scripts/run-tests.sh --concurrency 8 --url http://localhost/2283553-test-speed/ --class "Drupal\node\Tests\NodeAccessBaseTableTest","Drupal\node\Tests\NodeAccessFieldTest","Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest","Drupal\node\Tests\NodeAccessLanguageAwareTest","Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest"

Drupal test run
---------------

Tests to be run:
  - Drupal\node\Tests\NodeAccessBaseTableTest
  - Drupal\node\Tests\NodeAccessFieldTest
  - Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest
  - Drupal\node\Tests\NodeAccessLanguageAwareTest
  - Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest

Test run started:
  Sunday, August 10, 2014 - 01:31

Test summary
------------

Drupal\node\Tests\NodeAccessLanguageAwareTest                 91 passes                                      
Drupal\node\Tests\NodeAccessLanguageAwareCombinationTest     142 passes                                      
Drupal\node\Tests\MultiStepNodeFormBasicOptionsTest           17 passes                                      
Drupal\node\Tests\NodeAccessFieldTest                         50 passes                                      
Drupal\node\Tests\NodeAccessBaseTableTest                    159 passes                                      

Test run duration: 1 min 18 sec

Each child concurrently builds the environment so none can use the cached environment, it seems.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

droplet’s picture

droplet commented
Priority: Normal » Major
Status: Needs work » Needs review

I think the basic concept still valuable. Loading a SQL dump are so common in PHPUnit tests. There's maybe some concerns blocking this trick in d.org testbots. But I think at least we could provide an option for the developer to switch it on / off and capture particular DB snapshot.

Running an empty JavascriptTestBase test on Windows on a high-end PC. It took 47s.

Time: 46.83 seconds, Memory: 4.00Mb

A simple issue easily took an hour on little debugging: #2782915: Standardize the behavior of links when Outside In editing mode is enabled

As a developer, we understand what's going to test. A standard Drupal Installation bootstrap doesn't that important in most Patch Development & Re-testing on failing tests.

I wonder how possible to raise this issue to Critical Priority? Minimize Tests waiting time, we could focus more on patching.

** Although I thought it's not so right, I marked it as "Needs Review". A comment in this issue told me I can do it (, let me try if I could): #2747641: Add status "Needs Feedback" to issues

droplet’s picture

droplet commented
Priority: Major » Normal
Status: Needs review » Closed (duplicate)
Related issues: +#2747075: [meta] Improve WebTestCase / BrowserTestBase performance by 50%

Just see another issue trying to implement DB caching, pretty good movement tho: #2747075: [meta] Improve WebTestCase / BrowserTestBase performance by 50%