Warn/inform users when the hosting environment has a too low limit of APCU cache

+1 for warning. Drupal 8 is the first version to use a fixed-size cache back-end out of the box. Plenty of people used Memcached with D6 and D7, which was also fixed-size. But that was only for expert-level users, who knew what they're doing. Drupal 8 makes it too easy to shoot yourself in the foot:

  cache.bootstrap:
    class: Drupal\Core\Cache\CacheBackendInterface
    tags:
      - { name: cache.bin, default_backend: cache.backend.chainedfast }
    factory: cache_factory:get
    arguments: [bootstrap]
  cache.config:
    class: Drupal\Core\Cache\CacheBackendInterface
    tags:
      - { name: cache.bin, default_backend: cache.backend.chainedfast }
    factory: cache_factory:get
    arguments: [config]
…
  cache.discovery:
    class: Drupal\Core\Cache\CacheBackendInterface
    tags:
      - { name: cache.bin, default_backend: cache.backend.chainedfast }
    factory: cache_factory:get
    arguments: [discovery]

Plus this in \Drupal\Core\Cache\ChainedFastBackendFactory::__construct():

    // Default the fast backend to APCu if it's available.
    if (!isset($fast_service_name) && function_exists('apcu_fetch')) {
      $fast_service_name = 'cache.backend.apcu';
    }

Together, those mean that we'll default those 3 cache bins to use APCu (chained with DB) automatically. This is problematic as soon as you start having much config (bilingual effectively doubles the amount of config, trilingual triples it, et cetera).

I propose we make \Drupal\language\LanguageServiceProvider do slightly more: it's already making container changes based on whether a site is multilingual or not. I propose we also let it change cache.config's default_backend service tag.

I just discovered the related #2765271: Rationalize use of the 'discovery' cache bin, since it's stored in the limited size APCu by default. Should we do that first?

Also note that this was introduced in #2248767: Use fast, local cache back-end (APCu, if available) for low-write caches (bootstrap, discovery, and config). We assumed that all sites would have at least the default of 32 MB there.

+1

This code was heavily inspired by how we check the PHP memory limit. The language is equivalent as well.

APCu disabled

APCu enabled, with default limit

APCu disabled, low limit

It failed because it was rolled against 8.2.x. Now queued that one for testing against 8.2.x.

Attached is a dual reroll: one that's identical to #7 for 8.2.x, and a new patch that applies to 8.3.x. IMHO this should be committed to both 8.2.x and 8.3.x because it's valuable information regardless of version.

Testbot is having a bad day:

fail: [Other] Line 195 of core/modules/simpletest/src/InstallerTestBase.php:
Failed to set field driver to mysql

fail: [Other] Line 195 of core/modules/simpletest/src/InstallerTestBase.php:
Failed to set field mysql[username] to drupaltestbot

fail: [Other] Line 195 of core/modules/simpletest/src/InstallerTestBase.php:
Failed to set field mysql[password] to drupaltestbotpw

… or is this change preventing the installer from completing? I suspect it's that…

Thanks for confirming my suspicion in #14. Rerolled with fix.

Either testbot doesn't have APCu at all, or it has less than this recommended limit indeed.

The APCu settings on the testbot environments are as follows:

5.3:
apc.shm_size = 300M

5.5, 5.6, 7:
apc.shm_size = 2000M

But…

    // Provide a default configuration, if not set.
    if (!isset($configuration['default'])) {
      // @todo Use extension_loaded('apcu') for non-testbot
      //  https://www.drupal.org/node/2447753.
      if (function_exists('apcu_fetch')) {
        $configuration['default']['cache_backend_class'] = '\Drupal\Component\FileCache\ApcuFileCacheBackend';
      }
    }

Perhaps testbot has never been testing the APCu cache backend then?

So, let's debug this. Let's see what testbot really has.

Hum. I hacked run-test.sh in a way that works locally, but not on testbot apparently.

Testbot:

12:47:32 php /var/www/html/core/scripts/run-tests.sh  --list --php /opt/phpenv/shims/php > /var/www/html/artifacts/testgroups.txt
12:47:32 Command created as exec id 88eb3ef7
12:47:32 HTTP/1.1 200 OK
12:47:32 Content-Type: application/vnd.docker.raw-stream
12:47:32 
12:47:32 
12:47:32 Error
12:47:32 Received a non-zero return code from the last command executed on the container.  (Return status: 2)
12:47:32 Execution Error
12:47:32 Error encountered while executing job build step execute:command

Local:

$ php core/scripts/run-tests.sh
bool(true)
string(3) "64M"

or

$ php core/scripts/run-tests.sh --url http://d8 --class "\Drupal\page_cache\Tests\PageCacheTest"
bool(true)
string(3) "64M"

Anybody who knows run-tests.sh better who can reroll this?

Also, from the IS:

Observation: "Bigger" sites, with a lot of config, especially when its multilingual, pass by this limit. This leads to incredible slow sites, deadlock starts etc.

Opened #2832450: Multilingual config cached in "config" cache bin; quickly reaches APCu memory limits for that.

With a hint from @alexpott on IRC, I think this may be more successful.

• PHP 5.5 + MySQL 5.5: bool(true) + string(5) "2000M"
• PHP 5.5 + PostgreSQL 5.5: bool(true) + string(5) "2000M"
• PHP 5.5 + SQLite 3.8: bool(true) + string(5) "2000M"
• PHP 5.6 + MySQL 5.5: bool(true) + string(5) "2000M"
• PHP 7 + MySQL 5.5: bool(false) + bool(false)
• PHP 7 + PostgreSQL 9.1: bool(false) + bool(false)
• PHP 7 + SQLite 3.8: bool(false) + bool(false)

Conclusion: APCu is not enabled on PHP 7.

I am so very sorry. When I was looking at the apcu settings I missed that in the actual dockerfile for php7 there is this:

# TODO: test apcu on php 7
# RUN echo | pecl install channel://pecl.php.net/APCu-4.0.7
# TODO: no pecl yaml on php 7
# RUN pecl install yaml < /usr/lib/x86_64-linux-gnu

http://cgit.drupalcode.org/drupalci_environments/tree/web/web-7/Dockerfi...

When we launched drupalci, php7 was not yet out, and apcu did not work. This is remnants of that.

we should be addressing a new php7 container soon that includes apcu (and maybe 7.1?)

@Mixologic is there an issue to follow to know when this will be added to drupalci? Also, I see the todo is referencing 4.0.7 but to my knowledge the minimum version compatible with PHP7 is 5.1.0.

+    if (function_exists('apcu_fetch')) {
+      $requirements['php_apcu'] = array(
+        'value' => t('Not enabled'),
+        'severity' => REQUIREMENT_WARNING,
+        'description' => t('PHP APCu caching can improve your site\'s performance considerably. It is <strong>highly recommended</strong> to have <a href="http://php.net/manual/en/apcu.installation.php" target="_blank">APCu</a> installed on your server.'),
+      );
+    }

Don't we need to negate the condition?

Rerolled #16 and negated "function_exists" condition.

Applied short array syntax.

Fixed typo.

Slightly different approach. It checks available ACPu memory and warns if not much left. The severity is set dynamically.

Retesting, and also testing against PHP 7. That was the problem before.

Let's finish this.

1. +++ b/core/modules/system/system.install
   @@ -260,6 +260,38 @@ function system_requirements($phase) {
   +      $requirements['php_apcu']['description'] = t('Memory usage: %used of %total.', [
   +        '%used' => format_size($memory_info['seg_size'] - $memory_info['avail_mem']),
   +        '%total' => format_size($memory_info['seg_size']),
   +      ]);
   

   I'm not sure if this is enough information. I like that this is now dynamic, but I don't like that this removes so much information that was added in the patch Wim had earlier.

   It now no longer describes the default memory limit or the reason why that default is no longer sufficient.

2. +++ b/core/modules/system/system.install
   @@ -260,6 +260,38 @@ function system_requirements($phase) {
   +        'description' => t('PHP APCu caching can improve your site\'s performance considerably. It is <strong>highly recommended</strong> to have <a href="http://php.net/manual/en/apcu.installation.php" target="_blank">APCu</a> installed on your server.'),
   

   We don't usually add links in the actual translatable string, do we?

3. +++ b/core/modules/system/system.install
   @@ -260,6 +260,38 @@ function system_requirements($phase) {
   +
   +
   

   Should be only one empty line here.

So are you saying you'd like me to re-upload #16?

@Wim Leers not sure, I like the fact that there's more granular statuses in #39.

I think the ideal version would be to add the messages used in #16 to #39?

One minor issue we haven't considered yet is that APCu makes no sense in CLI. PHP may have different configurations for a web server and CLI. So that we should not report any warnings when the requirements are checked in CLI context (i.e. via drush rq command).

The patch brings together #38 and #39.

We don't usually add links in the actual translatable string, do we?

All php.net links in system_requirements() are hardcoded in translatable strings by some reason. I've used the same approach for consistency.

Slightly changed the description as the memory limit was duplicated.
Before:

After:

Maybe it's better to return back "Memory usage" description but remove memory limit value in in parentheses.

Overall this looks good to me. However,

1. I ran into a strange issue when playing around with the configuration values: if I set apcu.shm_size = 666K, it is listed as 666GB on the status page, which is obviously wrong:

   

   Verified on admin/reports/status/php, and there it is listed correctly as 666K:

   

   Specifying the size in Kb is valid according to https://www.php.net/manual/en/apc.configuration.php#ini.apc.shm-size. If I use 16000K as config value, the reported value is correct (15.62Mb). I think this is just a documentation issue, since when debugging this issue on the command line (to determine if this is a PHP bug), I get "PHP Warning: PHP Startup: apc.shm_size now uses M/G suffixes, please update your ini files in Unknown on line 0". Still, the fact that 16000K works and 666K does not might mean this *is* a PHP issue, not just documentation.

2. The logic in the patch is broken if I explictly disable apc using "apc.enabled=0" in my configuration. The output on the status page then becomes:

   

   In this case, extension_loaded('apcu') will return true (and so does function_exists('apcu_fetch')), so the best option might be to check ini_get('apc.enabled').

@mrbaileys
Re 1. It also produces weird output for me when using not M/G suffixes.

Still, the fact that 16000K works and 666K does not might mean this *is* a PHP issue, not just documentation.

That line may explain this behavior.
https://github.com/krakjoe/apcu/blob/v5.1.18/php_apc.c#L127

However, the value printed in the status report is taken from apcu_sma_info() which I believe returns a real value.

Re 2. Fixed.

Also added missing warning title when APCu is not enabled.

1. +++ b/core/modules/system/system.install
   @@ -312,6 +313,46 @@ function system_requirements($phase) {
   +    if (ini_get('apc.enabled')) {
   

   Probably still best to also include a function_exists() or extension_loaded() alongside the ini_get check. For reference, Symfony does:

   function_exists('apcu_fetch') && filter_var(ini_get('apc.enabled'), FILTER_VALIDATE_BOOLEAN)
   

   #2447753: APCu detection erroneously succeeds if apc.enabled is "0" also has a couple of variations on the check for APCu availability.

2. +++ b/core/modules/system/system.install
   @@ -312,6 +313,46 @@ function system_requirements($phase) {
   +        'description' => t('PHP APCu caching can improve your site\'s performance considerably. It is <strong>highly recommended</strong> to have <a href="http://php.net/manual/en/apcu.installation.php" target="_blank">APCu</a> installed on your server.'),
   

   I think it is preferrable to link to https://php.net instead of http://php.net.

The warning on the status page when APCu is unavalaible looks ok to me:

Addressed #58

I think it is preferable to link to https://php.net instead of http://php.net.

I think we need a ticket to fix all php.net links in Drupal code as we have tons of them.

This looks great to me!

Here are screenshots with the latest patch in #59:

Enabled/default config:

Disabled:

Enabled, low memory:

+++ b/core/modules/system/system.install
@@ -312,6 +313,46 @@ function system_requirements($phase) {
+      $requirements['php_apcu'] += [
+        'value' => t('Not enabled'),
+        'severity' => REQUIREMENT_WARNING,
+        'description' => t('PHP APCu caching can improve your site\'s performance considerably. It is <strong>highly recommended</strong> to have <a href="https://www.php.net/manual/apcu.installation.php" target="_blank">APCu</a> installed on your server.'),
+      ];

Making this a warning is a bit tricky. There is a situation where this warning could be unresolvable. If you are on a host with no control over your APCu being enabled, or if you can enable it you have no control over the available memory. If this is too small it would make sense to disable APCu and then you've still got a warning. For me this is a recommendation and should be REQUIREMENT_INFO. For me this is the same as using a PHP version which is not supported by PHP but still above the minimum version Drupal supports.

That said it could be argued that this the same opcache which we recommend with a warning. So maybe this is okay.

Addressed #63.

Note that current design of status report does not make any visual difference between REQUIREMENT_OK and REQUIREMENT_INFO.

Could we convert the actual size to MB too in the requirement message? The screenshot is making people compare 1023.88KB to 32MB.

No strong opinion between info and warning for this one.

Working on this...

Please review my patch as it has fix from comment #65.

+++ b/core/modules/system/system.install
@@ -316,6 +316,8 @@
+      $apcu_exploded = explode(' ', $apcu_actual_size);
+      $apcu_actual_size = $apcu_exploded[0] * 0.001 . ' MB';

I don't think this is going to work, since format_size() can return different sizes:

Your change assumes that format_size is always returning kilobytes. Additionally, the explode/division/concatination feels fragile.

I would actually suggest not altering the patch in #64: in most cases the shm_size will be set using MB (ex. 64 MB), and format_size will report it as 64 MB. It is only when configuring shm_size with non-standard values (such as 123456) that format_size will report with decimal.

#51 Still needs to be addressed.

$ drush rq
[warning] apcu_sma_info(): No APC SMA info available. Perhaps APC is disabled via apc.enabled? system.install:315
[warning] Division by zero system.install:316

@Chi: in #52 you added "... && PHP_SAPI != 'cli'" to the if-statement wrapping the requirements check, which seems to do the trick? I am not getting the "No APC SMA info available" warning when running drush rq...

Could we convert the actual size to MB too in the requirement message? The screenshot is making people compare 1023.88KB to 32MB.

apc.shm_size should always be configured in MB or GB. If you use another unit, php issues a warning: "Warning: PHP Startup: apc.shm_size now uses M/G suffixes, please update your ini files in Unknown on line 0". So if configured correctly, you will always see the output on the status screen reported in megabytes (see screenshot). The *only* exception I could find is when apc.shm_size is set to "1M", in which case it is reported as "1023.88 K". I don't think we should accommodate for this edge case, so tentatively setting back to RTBC.

@mr.baileys, you are right, I tested wrong patch.

apc.shm_size should always be configured in MB or GB. If you use another unit, php issues a warning

And furthermore the real memory size becomes unpredictable. See notes #55 and #56 for details.

Per #70 the current RTBC patch is in #64.

@Chi if we commit #64 then we've not addressed @catch's comment in #65 - ah #70 addresses this. I agree a 1mb APCu cache size is an edge case.

Committed and pushed cee0a33481 to 9.0.x and a1f442e6e3 to 8.9.x. Thanks!

I just created #3131358: Make APCu requirements errors precise and explanatory , which I believe is a follow-up to this issue.

Another follow-up.
#3142928: Status report wrongly warns of APCu memory limit when admin language is not English

In my experience, after updating Drupal to version 8.9.7 using language in Spanish (if this matter somehow).
The problem was this line:

$apcu_actual_size = $apcu_exploded[0] * 0.001 . ' MB';

I had 32 M dedicated to APC and the variable $apcu_exploded[0] already returns 32; so when the patch 2800605-apcu-warning-67.patch
was trying to multiply by 0.001 it was giving 0.0032 instead of 32 MB which is not much or at least is below the recommended.

I just leave it as:
$apcu_actual_size = $apcu_exploded[0] . ' MB';
create my own patch, apply it and everything works as expected.

Thanks
Walter Chaves

@wchaveslive please use issue from comment above yours to report it - #3142928: Status report wrongly warns of APCu memory limit when admin language is not English