Rationalize use of the 'discovery' cache bin, since it's stored in the limited size APCu by default

Note that this was introduced in #2248767: Use fast, local cache back-end (APCu, if available) for low-write caches (bootstrap, discovery, and config). We weren't aware then of problems with big cache entries.

Per default APCU has 257 slots for data

This is very interesting. http://php.net/manual/en/apc.configuration.php lists 4096 as the default.

I think the patch can be this simple.

But I think we need profiling to understand the impact of this before it can be committed.

Yes, definitely. Another option would be to get specific big cache entries out of the bin. config and discovery both have possibly very large amount of cache gets, and config can actually grow a lot too, if you have a lot of config.

Will try to collect some numbers from some of our bigger sites.

@dawehner did a blog post here which explains the 257 a bit more: http://dawehner.github.io/php/2016/07/13/osx-apc.html

On the patch itself, this bit is unnecessary:

+++ b/core/modules/system/system.install
@@ -1730,3 +1730,19 @@ function system_update_8201() {
+
+/**
+ * Rebuild the container to update the 'cache.discovery' service definition.
+ */
+function system_update_8300() {
+  // Empty update to cause a cache rebuild so that the container is rebuilt.
+}

The container gets rebuilt by core version number increments, no need for an update.

This potentially affects #2704571: Add an APCu classloader with a single entry too not just cache.discovery - would be good to know if it's an issue just with size or also with number, since that'd cut down the number a lot, but won't be small.

One way to approach this would be to gzip compress cache entries over a certain size (i.e. strlen() then compress with a flag to say it's compressed), but that's adding some complexity.

http://dawehner.github.io/php/2016/07/13/osx-apc.html is gold. Thanks.

I'm willing to bet 99% of people don't know about this "257 slots" thing.

Looked at one of our projects.
In total, I have 400 entries in the discovery bin pretty recently after a cache clear, on production in redis I see 440. the config cache bin on the other hand has 6k entries. So config is actually a bigger problem as far as number of entries goes.

The problem with discovery is that some entries can get pretty big. views_data is obviously a clear winner there. I thought I've opened an issue for that already but apparently not. Especially since views_data is something that you should *not* need at runtime, only when caches are built and in the backend. We could have two backends but the number of cache requests is likely fairly low.

token_info is already bug, this is because token.module now exposes tons of tokens for fields and properties by default.

From then on, it is mostly entity definitions and a few bigger plugin caches. We could move some of that too. But I think keeping most of the plugin definitions like all the views stuff in discovery and that in apcu would be beneficial.

mysql> select cid, length(data) from cache_discovery order by length(data) desc limit 30;                                                                                                   +----------------------------------------------------------------+--------------+
| cid                                                            | length(data) |
+----------------------------------------------------------------+--------------+
| views_data:de                                                  |      3208723 |
| token_info_sorted:de                                           |       820983 |
| entity_field_map                                               |       370954 |
| typed_config_definitions                                       |       310671 |
| typed_data_types_plugins                                       |       310201 |
| entity_field_storage_definitions:node:de                       |       190059 |
| local_task_plugins:de                                          |       189018 |
| entity_type                                                    |       174254 |
| block_plugins                                                  |       160135 |
| entity_field_storage_definitions:contact_message:de            |       135502 |
| library_info:zol                                               |        92346 |
| field_types_plugins                                            |        89483 |
| library_info:seven                                             |        82955 |
| entity_bundle_field_definitions:node:np8_news:de               |        54092 |
| entity_bundle_field_definitions:contact_message:zol_buy_abo:de |        46766 |
| views:argument_validator                                       |        45001 |
| rest_plugins                                                   |        44346 |
| views_data:file_managed:de                                     |        42595 |
| entity_bundle_field_definitions:node:np8_marketplace:de        |        42081 |
| element_info_build:seven                                       |        39027 |
| element_info_build:zol                                         |        38959 |
| entity_bundle_info:de                                          |        38280 |
| views_data:node_field_data:de                                  |        37940 |
| local_action_plugins:de                                        |        37755 |
| entity_bundle_field_definitions:node:zol_newspaper:de          |        35377 |
| entity_bundle_field_definitions:node:np8_local_organization:de |        35256 |
| views_data:taxonomy_term_field_data:de                         |        34529 |
| field_formatter_types_plugins                                  |        34419 |
| entity_bundle_field_definitions:node:np8_gallery:de            |        33850 |
| views:row                                                      |        28767 |
+----------------------------------------------------------------+--------------+

Edit, formatting doesn't work well it seems, better readable version: https://gist.github.com/Berdir/584a23ddd3169bccfb6c220d14b32018

The problem with discovery is that some entries can get pretty big.

Yep, exactly. This is also the analysis dawehner made in the IS.

Especially since views_data is something that you should *not* need at runtime, only when caches are built and in the backend.

So you are proposing to move all the views_data:* cache items out of the discovery cache bin, into a separate cache bin. Possibly just use the data cache bin?

Moving the main views data entry out is great.

I really doubt we need token in there either - most requests are not rendering tokens.

Should we open sub-issues to move those out and maybe make this a meta to rationalise what goes in there? That seems preferable to dropping chained_fast unless there's an insurmountable issue with apcu.

After that entity_field_map, typed_config_definitions and typed_data_types_plugins all look borderline, we need to look at where they get requested.

Should we open sub-issues to move those out and maybe make this a meta to rationalise what goes in there?

+1

Created #2824547: Change ViewsData to use the default cache bin instead of discovery and #2824548: Move token info cache to data cache bin

Both #2824547: Change ViewsData to use the default cache bin instead of discovery and #2824548: Move token info cache to data cache bin have landed! Both have been committed to 8.3.x only. Berdir's profiling at #2824548-4: Move token info cache to data cache bin looks very encouraging:

Before

mysql> select count(*), SUM(length(data)) / 1024 / 1024 as sum, max(length(data)) / 1024 / 1024 as max from cache_discovery;
+----------+------------+------------+
| count(*) | sum        | max        |
+----------+------------+------------+
|      425 | 9.20685387 | 3.06007671 |
+----------+------------+------------+

After

mysql> select count(*), SUM(length(data)) / 1024 / 1024 as sum, max(length(data)) / 1024 / 1024 as max from cache_discovery;
+----------+------------+------------+
| count(*) | sum        | max        |
+----------+------------+------------+
|      354 | 4.78700542 | 0.35403728 |
+----------+------------+------------+

Those two issues having been committed will reduce the chance of APCu errors on 8.3.x sites. But that doesn't help current production sites… so what about those?

Should we create a CR, to inform site owners of this change? It would only cause a very small percentage of site owners to look into things, but still, it's probably worth calling out.

From #16:

Should we open sub-issues to move those out and maybe make this a meta to rationalise what goes in there? That seems preferable to dropping chained_fast unless there's an insurmountable issue with apcu.

This is now done.

After that entity_field_map, typed_config_definitions and typed_data_types_plugins all look borderline, we need to look at where they get requested.

What do we want to do with those?

Opened #2832450: Multilingual config cached in "config" cache bin; quickly reaches APCu memory limits as the next step here.

@msonnabaum did some testing back in December 2016, before Drupal 8.3 was released. He wrote a script (attached).

Total user: 43.63MB

40908k - drupal.apcu_backend
1661k - drupal.class_loader
2105k - drupal.file_cache

apcu_backend by prefix:

3405k - discovery:entity_bundle_field_definitions:node
2563k - discovery:views_data:en
2243k - config:language:en
2088k - config:language:de
2047k - config:language:zh-hant
2031k - config:language:zh-hans
2028k - config:language:ja
2000k - config:language:fr
1997k - config:language:es
1965k - config:language:it
1926k - discovery:entity_field_storage_definitions:node
1212k - config:core:base_field_override
983k  - discovery:entity_bundle_field_definitions:taxonomy_term
882k  - config:views:view
620k  - config:field:field
560k  - discovery:token_info_sorted:de
560k  - discovery:token_info_sorted:en
560k  - discovery:token_info_sorted:es
560k  - discovery:token_info_sorted:it
560k  - discovery:token_info_sorted:zh-hans
560k  - discovery:token_info_sorted:zh-hant
560k  - discovery:token_info_sorted:fr
537k  - discovery:entity_bundle_field_definitions:paragraph
500k  - config:field:storage
405k  - discovery:views_data:node_field_data
319k  - discovery:entity_bundle_field_definitions:comment
302k  - config:core:entity_view_display
233k  - bootstrap:theme:active_theme
228k  - discovery:typed_config_definitions
205k  - discovery:typed_data_types_plugins
190k  - discovery:local_task_plugins:en
177k  - discovery:entity_base_field_definitions:node
169k  - discovery:entity_base_field_definitions:menu_link_content
145k  - discovery:entity_base_field_definitions:user
141k  - discovery:entity_base_field_definitions:taxonomy_term
141k  - discovery:views_data:views
137k  - discovery:entity_base_field_definitions:paragraph
135k  - discovery:local_task_plugins:zh-hant
131k  - discovery:local_task_plugins:it
131k  - discovery:entity_base_field_definitions:comment
129k  - discovery:entity_type
128k  - discovery:local_task_plugins:ja
128k  - discovery:local_task_plugins:zh-hans
128k  - discovery:local_task_plugins:es
123k  - discovery:entity_bundle_field_definitions:field_collection_item
121k  - discovery:entity_field_storage_definitions:paragraph
117k  - discovery:entity_field_storage_definitions:taxonomy_term
111k  - config:block:block
105k  - discovery:entity_base_field_definitions:block_content
98k   - discovery:library_info:gamepressbase
97k   - discovery:library_info:seven
94k   - discovery:views_data:taxonomy_term_field_data
94k   - config:language:content_settings

views_data we know is a problem that is getting addressed, but it looks like multilingual sites are getting hit pretty hard. I have a feeling that's possibly another bug that needs to be addressed in core.

I figured this would be a valuable data point here to continue this issue. It confirms the need for #2832450: Multilingual config cached in "config" cache bin; quickly reaches APCu memory limits.

Gzipping data effectively saves 80% size with minimal CPU cost. We use this compressed backend in redis successfully for quite some months. See
* Redis #2826332-2: Option to compress data
* Core #1281408: Add a compressing serializer decorator