At the moment we statically cache the result of listing config by prefix in CachedStorage::listAll(). We store this by prefix looked for. This patch takes a different approach to save database queries - it statically caches the entire list of config and then filters. In the standard profile this saves a query on every page due to blocks and tours.

CommentFileSizeAuthor
#1 2472561.1.patch4.93 KBalexpott

Comments

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Status: Active » Needs review
StatusFileSize
new 2472561.1.patch4.93 KB

Status: Needs review » Needs work

The last submitted patch, 1: 2472561.1.patch, failed testing.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented 
Drupal\simpletest\Tests\SimpleTestBrowserTest	103	1	0
Message	Group	Filename	Line	Function	Status
Requesting https.php with a legitimate simpletest User-Agent returns OK.	Browser	SimpleTestBrowserTest.php	105	Drupal\simpletest\Tests\SimpleTestBrowserTest->testUserAgentValidation()

Werd. Does not fail locally.

Status: Needs work » Needs review

alexpott queued 1: 2472561.1.patch for re-testing.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented 
### FINAL REPORT

=== baseline-8.0.x..8.0.x compared (552ffd23adb6f..552ffd8a03bf7):

ct  : 53,540|53,540|0|0.0%
wt  : 718,745|856,389|137,644|19.2%
mu  : 41,307,448|41,308,472|1,024|0.0%
pmu : 41,377,968|41,379,576|1,608|0.0%

=== baseline-8.0.x..2472561 compared (552ffd23adb6f..552ffe63d4651):

ct  : 53,540|54,418|878|1.6%
wt  : 718,745|795,241|76,496|10.6%
mu  : 41,307,448|41,338,704|31,256|0.1%
pmu : 41,377,968|41,410,248|32,280|0.1%

---
ct = function calls, wt = wall time, cpu = cpu time used, mu = memory usage, pmu = peak memory usage

### XHPROF-LIB REPORT

+-------------------+------------+------------+------------+------------+------------+
| namespace         |        min |        max |       mean |     median |       95th |
+-------------------+------------+------------+------------+------------+------------+
| Calls             |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           |     54,418 |     57,138 |     54,445 |     54,418 |     54,418 |
| 8_0_x             |     53,540 |     56,260 |     53,567 |     53,540 |     53,540 |
|                   |            |            |            |            |            |
| Wall time         |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           |    795,241 |  2,002,876 |  1,186,935 |  1,066,496 |  1,762,056 |
| 8_0_x             |    856,389 |  1,970,084 |  1,296,479 |  1,234,643 |  1,836,134 |
|                   |            |            |            |            |            |
| Memory usage      |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           | 41,337,656 | 41,376,216 | 41,338,765 | 41,338,704 | 41,338,704 |
| 8_0_x             | 41,306,256 | 41,346,368 | 41,308,223 | 41,307,800 | 41,308,472 |
|                   |            |            |            |            |            |
| Peak memory usage |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           | 41,408,464 | 41,448,736 | 41,409,974 | 41,410,248 | 41,410,248 |
| 8_0_x             | 41,377,280 | 41,418,688 | 41,379,076 | 41,378,936 | 41,379,576 |
|                   |            |            |            |            |            |
+-------------------+------------+------------+------------+------------+------------+

So this looks like a significant saving. This benchmark is with render cache disabled.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented 
### FINAL REPORT

=== baseline-8.0.x..8.0.x compared (553003a5b7fe9..553003eb15b8d):

ct  : 53,540|53,540|0|0.0%
wt  : 328,294|325,060|-3,234|-1.0%
mu  : 41,306,936|41,308,472|1,536|0.0%
pmu : 41,377,720|41,379,576|1,856|0.0%

=== baseline-8.0.x..2472561 compared (553003a5b7fe9..5530042c1e3c4):

ct  : 53,540|54,418|878|1.6%
wt  : 328,294|322,370|-5,924|-1.8%
mu  : 41,306,936|41,337,976|31,040|0.1%
pmu : 41,377,720|41,408,672|30,952|0.1%

---
ct = function calls, wt = wall time, cpu = cpu time used, mu = memory usage, pmu = peak memory usage

### XHPROF-LIB REPORT

+-------------------+------------+------------+------------+------------+------------+
| namespace         |        min |        max |       mean |     median |       95th |
+-------------------+------------+------------+------------+------------+------------+
| Calls             |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           |     54,418 |     57,138 |     54,445 |     54,418 |     54,418 |
| 8_0_x             |     53,540 |     56,335 |     53,568 |     53,540 |     53,540 |
|                   |            |            |            |            |            |
| Wall time         |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           |    322,370 |    358,100 |    337,748 |    337,837 |    346,405 |
| 8_0_x             |    325,060 |    377,359 |    336,787 |    336,257 |    345,300 |
|                   |            |            |            |            |            |
| Memory usage      |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           | 41,337,656 | 41,375,696 | 41,338,704 | 41,338,340 | 41,338,704 |
| 8_0_x             | 41,306,960 | 41,350,656 | 41,308,564 | 41,308,472 | 41,308,472 |
|                   |            |            |            |            |            |
| Peak memory usage |            |            |            |            |            |
|                   |            |            |            |            |            |
| 2472561           | 41,408,464 | 41,448,640 | 41,409,842 | 41,409,460 | 41,410,248 |
| 8_0_x             | 41,377,856 | 41,551,448 | 41,380,781 | 41,379,576 | 41,379,576 |
|                   |            |            |            |            |            |
+-------------------+------------+------------+------------+------------+------------+

Looks like something heavy was running in the background - the results from #5 can not be trusted.

We save a query but the savings don't look like much.

berdir’s picture

berdir
Primary language German
Location Switzerland
commented

This makes sense.

* But doesn't that report actually say that it is 1.6% slower? Doing some specific micro-benchmarking might be more useful here, compare doing 2-3 separate queries vs. one + filtering in PHP. The thing is that mysql has an index on that table and is probably actually faster in filtering, but there's network overhead of course.
* I'm wondering is how much overhead it is to get all config names. I currently have 1500 config objects...
* We should update the unit test to return a number of config names that don't match, to test the filtering...
* We got rid of actually caching this list due to the expensive cache tag invalidation that happpened lots of times, especially during repeated cache writes like installation and config sync. However, if we only have a single cache key, and can maintain it during a single process, it might be almost worth considering to introduce that again. Definitely in a separate issue, and we need to profile how much difference it actually makes (apc_fetch() + unserialize of 1500 array values or a database query to fetch them). And we need to protect against concurrent writes.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

I'm wondering is how much overhead it is to get all config names. I currently have 1500 config objects...

I also asked Alex this very question, but can't remember his answer.

fabianx’s picture

fabianx commented

I also get very kinda the same, but more function calls numbers:

### FINAL REPORT

=== 8.0.x..8.0.x compared (553101a117fbd..553107ae86c51):

ct  : 54,417|54,417|0|0.0%
wt  : 91,701|92,830|1,129|1.2%
mu  : 19,751,184|19,751,184|0|0.0%
pmu : 19,875,576|19,875,576|0|0.0%

=== 8.0.x..issue-2472561--static-config compared (553101a117fbd..5531085170c34):

ct  : 54,417|55,289|872|1.6%
wt  : 91,701|93,355|1,654|1.8%
mu  : 19,751,184|19,782,800|31,616|0.2%
pmu : 19,875,576|19,907,264|31,688|0.2%

=== SUM: 8_0_x-summary..issue-2472561--static-config-summary compared (5531090229476..5531093dc61b9):

ct  : 54,419,722|55,292,867|873,145|1.6%
wt  : 102,769,737|103,312,884|543,147|0.5%
mu  : 19,752,461,824|19,783,285,808|30,823,984|0.2%
pmu : 19,876,883,768|19,907,774,040|30,890,272|0.2%

---
ct = function calls, wt = wall time, cpu = cpu time used, mu = memory usage, pmu = peak memory usage

### XHPROF-LIB REPORT

+------------------------------+------------+------------+------------+------------+------------+
| namespace                    |        min |        max |       mean |     median |       95th |
+------------------------------+------------+------------+------------+------------+------------+
| Calls                        |            |            |            |            |            |
|                              |            |            |            |            |            |
| issue-2472561--static-config |     55,289 |     58,011 |     55,293 |     55,289 |     55,289 |
| 8_0_x                        |     54,417 |     57,139 |     54,420 |     54,417 |     54,417 |
|                              |            |            |            |            |            |
| Wall time                    |            |            |            |            |            |
|                              |            |            |            |            |            |
| issue-2472561--static-config |     93,355 |    143,188 |    103,313 |    103,421 |    108,200 |
| 8_0_x                        |     92,830 |    145,153 |    102,770 |    103,051 |    107,533 |
|                              |            |            |            |            |            |
| Memory usage                 |            |            |            |            |            |
|                              |            |            |            |            |            |
| issue-2472561--static-config | 19,779,544 | 20,222,776 | 19,783,286 | 19,782,800 | 19,782,912 |
| 8_0_x                        | 19,751,184 | 20,193,976 | 19,752,462 | 19,751,184 | 19,751,192 |
|                              |            |            |            |            |            |
| Peak memory usage            |            |            |            |            |            |
|                              |            |            |            |            |            |
| issue-2472561--static-config | 19,904,080 | 20,354,872 | 19,907,774 | 19,907,264 | 19,907,296 |
| 8_0_x                        | 19,875,576 | 20,326,344 | 19,876,884 | 19,875,576 | 19,875,888 |
|                              |            |            |            |            |            |
+------------------------------+------------+------------+------------+------------+------------+

So I agree with #7 that we need some micro benchmarks here.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Issue tags: +Configuration system
berdir’s picture

berdir
Primary language German
Location Switzerland
commented
Status: Needs review » Needs work

Setting to needs work then for some of my feedback.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev
wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Is this still relevant?

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.