Problem/Motivation

In #2765271: Rationalize use of the 'discovery' cache bin, since it's stored in the limited size APCu by default, we're rationalizing the use of the discovery cache bin.

We already landed #2824547: Change ViewsData to use the default cache bin instead of discovery and #2824548: Move token info cache to data cache bin as part of that. That really helps, see #2765271-19: Rationalize use of the 'discovery' cache bin, since it's stored in the limited size APCu by default.

But, multilingual sites face a bigger problem: multilingual (translated) configuration. (@dawehner already mentioned this in the issue summary of #2800605: Warn/inform users when the hosting environment has a too low limit of APCU cache.) Because

  config.typed:
    class: Drupal\Core\Config\TypedConfigManager
    arguments: ['@config.storage', '@config.storage.schema', '@cache.discovery', '@module_handler']

means that all translated configuration is also cached in the discovery bin. So, the more multilingual you are, the bigger of a problem you have.

This can easily result in a few megabytes per language … meaning that you can reach the APCu limit rather quickly.

Proposed resolution

TBD

Remaining tasks

TBD

User interface changes

None.

API changes

None.

Data model changes

None.

Comments

Wim Leers created an issue. See original summary.

Wim Leers’s picture

Berdir’s picture

Uhm, no? :)

That is the config schema plugin manager. It has the config schema definitions, not actual config.

Yes, translated config is in apcu, like all other config too, but as part of the config cache bin?

Gábor Hojtsy’s picture

Right, if we are concerned for multilingual config, then this snippet in the issue summary is not correct, that does indeed involve the schema:

  config.typed:
    class: Drupal\Core\Config\TypedConfigManager
    arguments: ['@config.storage', '@config.storage.schema', '@cache.discovery', '@module_handler']

Unless of course we cache it translated(?) -- but I doubt that.

Berdir’s picture

Status: Active » Postponed (maintainer needs more info)
stella’s picture

I can confirm that we've started getting APCu cache limit problems as soon as we started adding in multilingual content and config. So much so that the errors filled up our log files and we ran out of space. Thankfully not live yet!

stella’s picture

Oh and by the way, we will have 13 languages in total (including en-us, en-gb, etc), so this is a big issue for us. More than happy to test patches or do whatever I can to help on this.

Berdir’s picture

@stella:

Right.. but your problem isn't going to be the config *schema* cache.. if anything it's going to be the actual config translations.

Berdir’s picture

Title: Multilingual config cached in "discovery" cache bin; quickly reaches APCu memory limits » Multilingual config cached in "config" cache bin; quickly reaches APCu memory limits
Status: Postponed (maintainer needs more info) » Active

Updating title based on how understand this, has nothing to do with discovery cache bin but with config. Still needs an issue summary update.

Also discussed with @stella in IRC. The default of having config in fast chained in config makes sense for no multlingual, it probably still makes sense for 1-2 additional languages.. for 13 languages, probably not :)

As a workaround, don't put it in fast chained: $settings['cache']['bins']['config'] = 'cache.backend.database'; (or redis, or memcache)

I'm not sure what we can do about this in core. I don't like not putting it in fast chained, I don't like not caching non-default collections although it might not make a big difference at least when using database backend.

effulgentsia’s picture

Not sure if it's a good idea or feasible, but maybe allowing something in $settings (or services.yml) to specify different cache backends for different config collections?

effulgentsia’s picture

Or similarly, for config collections to be assignable to a different cache bin?

Berdir’s picture

Maybe, but if you do have a multilingual, then every config load will always also have to check for the existense of a language override, so you'll have just as many config reads for collection as not.

Hm.. I guess there is a pretty large percentage of configs that do *not* exist in language collections. Maybe we could cache non-existing config in a single global blacklist-cache instead of creating hundreds of cache entries that just contain FALSE as we do right now?

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

anavarre’s picture

Is there any hope to get this in 8.3.x or at this point should we directly move to 8.4.x?

Gábor Hojtsy’s picture

Category: Task » Bug report

As per https://www.drupal.org/core/d8-allowed-changes#beta among things allowed in beta are bug fixes and contributed project blockers, if they are non-disruptive, or if the impact outweighs the disruption. I would argue if the APC limits are reached this is a bug not a task. Then its up to committers to asses the disruption vs. impact. I think a patch if possible would help asses that.

anavarre’s picture

Version: 8.3.x-dev » 8.4.x-dev

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Wim Leers’s picture

Quoting #2765271-24: Rationalize use of the 'discovery' cache bin, since it's stored in the limited size APCu by default:

@msonnabaum did some testing back in December 2016, before Drupal 8.3 was released. He wrote a script (attached).

Total user: 43.63MB

40908k - drupal.apcu_backend
1661k - drupal.class_loader
2105k - drupal.file_cache

apcu_backend by prefix:

3405k - discovery:entity_bundle_field_definitions:node
2563k - discovery:views_data:en
2243k - config:language:en
2088k - config:language:de
2047k - config:language:zh-hant
2031k - config:language:zh-hans
2028k - config:language:ja
2000k - config:language:fr
1997k - config:language:es
1965k - config:language:it
1926k - discovery:entity_field_storage_definitions:node
1212k - config:core:base_field_override
983k  - discovery:entity_bundle_field_definitions:taxonomy_term
882k  - config:views:view
620k  - config:field:field
560k  - discovery:token_info_sorted:de
560k  - discovery:token_info_sorted:en
560k  - discovery:token_info_sorted:es
560k  - discovery:token_info_sorted:it
560k  - discovery:token_info_sorted:zh-hans
560k  - discovery:token_info_sorted:zh-hant
560k  - discovery:token_info_sorted:fr
537k  - discovery:entity_bundle_field_definitions:paragraph
500k  - config:field:storage
405k  - discovery:views_data:node_field_data
319k  - discovery:entity_bundle_field_definitions:comment
302k  - config:core:entity_view_display
233k  - bootstrap:theme:active_theme
228k  - discovery:typed_config_definitions
205k  - discovery:typed_data_types_plugins
190k  - discovery:local_task_plugins:en
177k  - discovery:entity_base_field_definitions:node
169k  - discovery:entity_base_field_definitions:menu_link_content
145k  - discovery:entity_base_field_definitions:user
141k  - discovery:entity_base_field_definitions:taxonomy_term
141k  - discovery:views_data:views
137k  - discovery:entity_base_field_definitions:paragraph
135k  - discovery:local_task_plugins:zh-hant
131k  - discovery:local_task_plugins:it
131k  - discovery:entity_base_field_definitions:comment
129k  - discovery:entity_type
128k  - discovery:local_task_plugins:ja
128k  - discovery:local_task_plugins:zh-hans
128k  - discovery:local_task_plugins:es
123k  - discovery:entity_bundle_field_definitions:field_collection_item
121k  - discovery:entity_field_storage_definitions:paragraph
117k  - discovery:entity_field_storage_definitions:taxonomy_term
111k  - config:block:block
105k  - discovery:entity_base_field_definitions:block_content
98k   - discovery:library_info:gamepressbase
97k   - discovery:library_info:seven
94k   - discovery:views_data:taxonomy_term_field_data
94k   - config:language:content_settings

views_data we know is a problem that is getting addressed, but it looks like multilingual sites are getting hit pretty hard. I have a feeling that's possibly another bug that needs to be addressed in core.

I figured this would be a valuable data point here to continue this issue. It confirms the need for #2832450: Multilingual config cached in "config" cache bin; quickly reaches APCu memory limits.

swentel’s picture

but it looks like multilingual sites are getting hit pretty hard

Yeah, we have a site with 46 languages, we had to set this default value pretty high to get it running without any problems.

anavarre’s picture

@swentel could you please try Mark's script and report back with data here? With 46 languages you'll likely be at the very edge of the spectrum, which would give a good indication of where we need to go with this issue.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.