At the moment we statically cache the result of listing config by prefix in CachedStorage::listAll(). We store this by prefix looked for. This patch takes a different approach to save database queries - it statically caches the entire list of config and then filters. In the standard profile this saves a query on every page due to blocks and tours.

Files: 
CommentFileSizeAuthor
#1 2472561.1.patch4.93 KBalexpott
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 91,539 pass(es). View

Comments

alexpott’s picture

Status: Active » Needs review
FileSize
4.93 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 91,539 pass(es). View

Status: Needs review » Needs work

The last submitted patch, 1: 2472561.1.patch, failed testing.

alexpott’s picture

Drupal\simpletest\Tests\SimpleTestBrowserTest	103	1	0
Message	Group	Filename	Line	Function	Status
Requesting https.php with a legitimate simpletest User-Agent returns OK.	Browser	SimpleTestBrowserTest.php	105	Drupal\simpletest\Tests\SimpleTestBrowserTest->testUserAgentValidation()

Werd. Does not fail locally.

Status: Needs work » Needs review

alexpott queued 1: 2472561.1.patch for re-testing.

alexpott’s picture

### FINAL REPORT

=== baseline-8.0.x..8.0.x compared (552ffd23adb6f..552ffd8a03bf7):

ct  : 53,540|53,540|0|0.0%
wt  : 718,745|856,389|137,644|19.2%
mu  : 41,307,448|41,308,472|1,024|0.0%
pmu : 41,377,968|41,379,576|1,608|0.0%

=== baseline-8.0.x..2472561 compared (552ffd23adb6f..552ffe63d4651):

ct  : 53,540|54,418|878|1.6%
wt  : 718,745|795,241|76,496|10.6%
mu  : 41,307,448|41,338,704|31,256|0.1%
pmu : 41,377,968|41,410,248|32,280|0.1%

---
ct = function calls, wt = wall time, cpu = cpu time used, mu = memory usage, pmu = peak memory usage

### XHPROF-LIB REPORT

+-------------------+------------+------------+------------+------------+------------+
| namespace         |        min |        max |       mean |     median |       95th |
+-------------------+------------+------------+------------+------------+------------+
| Calls             |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           |     54,418 |     57,138 |     54,445 |     54,418 |     54,418 |
| 8_0_x             |     53,540 |     56,260 |     53,567 |     53,540 |     53,540 |
|                   |            |            |            |            |            |
| Wall time         |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           |    795,241 |  2,002,876 |  1,186,935 |  1,066,496 |  1,762,056 |
| 8_0_x             |    856,389 |  1,970,084 |  1,296,479 |  1,234,643 |  1,836,134 |
|                   |            |            |            |            |            |
| Memory usage      |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           | 41,337,656 | 41,376,216 | 41,338,765 | 41,338,704 | 41,338,704 |
| 8_0_x             | 41,306,256 | 41,346,368 | 41,308,223 | 41,307,800 | 41,308,472 |
|                   |            |            |            |            |            |
| Peak memory usage |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           | 41,408,464 | 41,448,736 | 41,409,974 | 41,410,248 | 41,410,248 |
| 8_0_x             | 41,377,280 | 41,418,688 | 41,379,076 | 41,378,936 | 41,379,576 |
|                   |            |            |            |            |            |
+-------------------+------------+------------+------------+------------+------------+

So this looks like a significant saving. This benchmark is with render cache disabled.

alexpott’s picture

### FINAL REPORT

=== baseline-8.0.x..8.0.x compared (553003a5b7fe9..553003eb15b8d):

ct  : 53,540|53,540|0|0.0%
wt  : 328,294|325,060|-3,234|-1.0%
mu  : 41,306,936|41,308,472|1,536|0.0%
pmu : 41,377,720|41,379,576|1,856|0.0%

=== baseline-8.0.x..2472561 compared (553003a5b7fe9..5530042c1e3c4):

ct  : 53,540|54,418|878|1.6%
wt  : 328,294|322,370|-5,924|-1.8%
mu  : 41,306,936|41,337,976|31,040|0.1%
pmu : 41,377,720|41,408,672|30,952|0.1%

---
ct = function calls, wt = wall time, cpu = cpu time used, mu = memory usage, pmu = peak memory usage

### XHPROF-LIB REPORT

+-------------------+------------+------------+------------+------------+------------+
| namespace         |        min |        max |       mean |     median |       95th |
+-------------------+------------+------------+------------+------------+------------+
| Calls             |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           |     54,418 |     57,138 |     54,445 |     54,418 |     54,418 |
| 8_0_x             |     53,540 |     56,335 |     53,568 |     53,540 |     53,540 |
|                   |            |            |            |            |            |
| Wall time         |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           |    322,370 |    358,100 |    337,748 |    337,837 |    346,405 |
| 8_0_x             |    325,060 |    377,359 |    336,787 |    336,257 |    345,300 |
|                   |            |            |            |            |            |
| Memory usage      |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           | 41,337,656 | 41,375,696 | 41,338,704 | 41,338,340 | 41,338,704 |
| 8_0_x             | 41,306,960 | 41,350,656 | 41,308,564 | 41,308,472 | 41,308,472 |
|                   |            |            |            |            |            |
| Peak memory usage |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           | 41,408,464 | 41,448,640 | 41,409,842 | 41,409,460 | 41,410,248 |
| 8_0_x             | 41,377,856 | 41,551,448 | 41,380,781 | 41,379,576 | 41,379,576 |
|                   |            |            |            |            |            |
+-------------------+------------+------------+------------+------------+------------+

Looks like something heavy was running in the background - the results from #5 can not be trusted.

We save a query but the savings don't look like much.

Berdir’s picture

This makes sense.

* But doesn't that report actually say that it is 1.6% slower? Doing some specific micro-benchmarking might be more useful here, compare doing 2-3 separate queries vs. one + filtering in PHP. The thing is that mysql has an index on that table and is probably actually faster in filtering, but there's network overhead of course.
* I'm wondering is how much overhead it is to get all config names. I currently have 1500 config objects...
* We should update the unit test to return a number of config names that don't match, to test the filtering...
* We got rid of actually caching this list due to the expensive cache tag invalidation that happpened lots of times, especially during repeated cache writes like installation and config sync. However, if we only have a single cache key, and can maintain it during a single process, it might be almost worth considering to introduce that again. Definitely in a separate issue, and we need to profile how much difference it actually makes (apc_fetch() + unserialize of 1500 array values or a database query to fetch them). And we need to protect against concurrent writes.

Wim Leers’s picture

I'm wondering is how much overhead it is to get all config names. I currently have 1500 config objects...

I also asked Alex this very question, but can't remember his answer.

Fabianx’s picture

I also get very kinda the same, but more function calls numbers:

### FINAL REPORT

=== 8.0.x..8.0.x compared (553101a117fbd..553107ae86c51):

ct  : 54,417|54,417|0|0.0%
wt  : 91,701|92,830|1,129|1.2%
mu  : 19,751,184|19,751,184|0|0.0%
pmu : 19,875,576|19,875,576|0|0.0%

=== 8.0.x..issue-2472561--static-config compared (553101a117fbd..5531085170c34):

ct  : 54,417|55,289|872|1.6%
wt  : 91,701|93,355|1,654|1.8%
mu  : 19,751,184|19,782,800|31,616|0.2%
pmu : 19,875,576|19,907,264|31,688|0.2%

=== SUM: 8_0_x-summary..issue-2472561--static-config-summary compared (5531090229476..5531093dc61b9):

ct  : 54,419,722|55,292,867|873,145|1.6%
wt  : 102,769,737|103,312,884|543,147|0.5%
mu  : 19,752,461,824|19,783,285,808|30,823,984|0.2%
pmu : 19,876,883,768|19,907,774,040|30,890,272|0.2%

---
ct = function calls, wt = wall time, cpu = cpu time used, mu = memory usage, pmu = peak memory usage

### XHPROF-LIB REPORT

+------------------------------+------------+------------+------------+------------+------------+
| namespace                    |        min |        max |       mean |     median |       95th |
+------------------------------+------------+------------+------------+------------+------------+
| Calls                        |            |            |            |            |            |
|                              |            |            |            |            |            |
| issue-2472561--static-config |     55,289 |     58,011 |     55,293 |     55,289 |     55,289 |
| 8_0_x                        |     54,417 |     57,139 |     54,420 |     54,417 |     54,417 |
|                              |            |            |            |            |            |
| Wall time                    |            |            |            |            |            |
|                              |            |            |            |            |            |
| issue-2472561--static-config |     93,355 |    143,188 |    103,313 |    103,421 |    108,200 |
| 8_0_x                        |     92,830 |    145,153 |    102,770 |    103,051 |    107,533 |
|                              |            |            |            |            |            |
| Memory usage                 |            |            |            |            |            |
|                              |            |            |            |            |            |
| issue-2472561--static-config | 19,779,544 | 20,222,776 | 19,783,286 | 19,782,800 | 19,782,912 |
| 8_0_x                        | 19,751,184 | 20,193,976 | 19,752,462 | 19,751,184 | 19,751,192 |
|                              |            |            |            |            |            |
| Peak memory usage            |            |            |            |            |            |
|                              |            |            |            |            |            |
| issue-2472561--static-config | 19,904,080 | 20,354,872 | 19,907,774 | 19,907,264 | 19,907,296 |
| 8_0_x                        | 19,875,576 | 20,326,344 | 19,876,884 | 19,875,576 | 19,875,888 |
|                              |            |            |            |            |            |
+------------------------------+------------+------------+------------+------------+------------+

So I agree with #7 that we need some micro benchmarks here.

alexpott’s picture

Issue tags: +Configuration system
Berdir’s picture

Status: Needs review » Needs work

Setting to needs work then for some of my feedback.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.