APCu cache key corruption causes installation failures [#2488730]

Problem/Motivation

Drupal GUI install fails on PHP 5.5 with certain commonly shipped versions of APCu. This is due to a bug in early versions in APCu, not in Drupal. Versions of APCu of 4.0.7 and later are known to be safe.

Originally, when this issue was encountered, Drupal 8 had a minimum PHP version of 5.4. This has since been raised to 5.5.9 (issue #2508231). It is believed that the faulty APCu was no longer being distributed by the time PHP 5.5.9 was released. Therefore, this issue has been closed.

Observed Symptoms

During a GUI install of Drupal 8 (either standard or minimal) exceptions are thrown from \Drupal\Component\FileCache\FileCache::getMultiple(). These are:reporting missing elements in $cached: either $cached['mtime'] or $cached['filepath'].
On Safari these cause the installation to stop. On Firefox a page refresh causes the installation to continue without obvious errors, but the installation is faulty resulting in a large number of unit test or other failures.

Notice: Undefined index: mtime in Drupal\Component\FileCache\FileCache->getMultiple() (line 115 of /Users/rob/Sites/drupalcon/drupaltemp/core/lib/Drupal/Component/FileCache/FileCache.php).
Drupal\Component\FileCache\FileCache->getMultiple(Array)
Drupal\Component\FileCache\FileCache->get('core/core.services.yml')
Drupal\Core\DependencyInjection\YamlFileLoader->load('core/core.services.yml')
Drupal\Core\DrupalKernel->compileContainer()
Drupal\Core\DrupalKernel->initializeContainer(1)
Drupal\Core\Installer\InstallerKernel->initializeContainer()
Drupal\Core\DrupalKernel->boot()
install_begin_request(Object, Array)
install_drupal(Object)

Environment

The environment being used for the install was:

OS: MacOS 10.10.3
Stack: MAMP 2.2
PHP: 5.5.3
APCu: 4.0.1

Detail

This has been traced back to key corruption in the $cached array. In one example the 'mtime' key has been replaced with '/[^A-' and the 'filepath' key has been replaced with 'original'. The values are as expected.

Cause Hypothesis

Alex Pott suggested that this might be caused by APCu serialisation being faulty and only working properly with APC. APC was replaced with APCu in PHP 5.5. He suggested that upgrading APCu to the latest version might fix the problem, but this turned out to be very difficult.

Proposed Resolution

The proposed resolution is to disable APC/APCu when using PHP >=5.5 and APCu < 4.0.7. A helper function in \Drupal\Component\Utility\Environment is used for detecting whether a suitable version of APC or APCu is installed, because it is known that this will help with fixing other specific caching issues in the future.
The three places where APC/APCu is detected with function_exists('apc_fetch') are replaced with calls to Environment::checkForCompatibleApcUserCache().

Comment	File	Size	Author
#17	2488730-17-check-for-apcu-version.patch	5.44 KB	blackra
#15	2488730-14-check-for-apcu-version.patch	5.51 KB	blackra
#12	2488730-12-check-for-apcu-version.patch	5.49 KB	blackra
#9	2488730-9-check-for-apcu-version.patch	4.12 KB	blackra
#6	2488730-6-check-for-apcu-version.patch	2.88 KB	blackra
#4	2488730-4-check-for-apcu-version.patch	1.33 KB	blackra
	apcu2-screenshot.png	473.13 KB	blackra

Comments

Comment #1

blackra commented 14 May 2015 at 18:17

PHP = 5.5.21, APC = 4.07 seems to fix this problem.

Comment #2

fabianx commented 14 May 2015 at 19:19

https://github.com/krakjoe/apcu/issues/35

This is the upstream issue for the bug, its an APC bug.

We should warn if version is prone to corruption, leaving active for that.

Comment #3

dawehner

German

commented 14 May 2015 at 20:23

We should warn if version is prone to corruption, leaving active for that.

That sounds like a great idea. Do we know which version causes the problem? For now our bisect range is 4.0.1 has issues 4.0.7 doesn't.

Comment #4

blackra commented 14 May 2015 at 20:48

Status:

Active

» Needs review

Status	File	Size
new	2488730-4-check-for-apcu-version.patch	1.33 KB

We don't know exactly which versions are affected so I have rolled a patch based on the following assumptions:

The bug exists in APCu versions < 4.0.7
The bug does not exist in APC.

We could do something more sophisticated if we checked which serializer was in use, but I'm not sure whether it is worth it. The key point of this patch is to avoid affecting people who are not capable of upgrading their environment. I think it would be reasonable to assume that people who care about performance will find a way to fix it.

Do we need to flag this as an issue in the system status screen?

Comment #5

fabianx commented 15 May 2015 at 04:18

Status:

Needs review

» Needs work

The patch is a great idea, but there is more cases in Drupal where we use function_exists('apc_fetch').

We should consolidate those into a helper class in Drupal\Component\Utility\Requirements and re-use from all places ...

That will also make a related issue easier ...

Thanks!

Comment #6

blackra commented 15 May 2015 at 16:46

Status:

Needs work

» Needs review

Status	File	Size
new	2488730-6-check-for-apcu-version.patch	2.88 KB

Rolled a new patch. There is no Requirements. I assumed you meant Environment. If not, I can move it.

Note that I deleted your @todo about the testbot. Is that still relevant?

Comment #7

dawehner

German

commented 15 May 2015 at 18:26

+++ b/core/lib/Drupal/Component/Utility/Environment.php
@@ -42,4 +42,54 @@ public static function checkMemoryLimit($required, $memory_limit = NULL) {
+  public static function checkForOriginalApc() {

Do we need that as a public method?

+++ b/core/lib/Drupal/Component/Utility/Environment.php
@@ -42,4 +42,54 @@ public static function checkMemoryLimit($required, $memory_limit = NULL) {
+    $compatibleApc = false;

we don't use camelcase for local variables.

+++ b/core/lib/Drupal/Component/Utility/Environment.php
@@ -42,4 +42,54 @@ public static function checkMemoryLimit($required, $memory_limit = NULL) {
+      $apcu_ext = new \ReflectionExtension('apcu');
+      $apcu_version = $apcu_ext->getVersion();
+      $compatibleApc = (version_compare($apcu_version, '4.0.7') >= 0);

Maybe its worth to static cache that.

+++ b/core/lib/Drupal/Core/DrupalKernel.php
@@ -425,9 +425,8 @@ public function boot() {
+      if (\Drupal\Component\Utility\Environment::checkForCompatibleApcUserCache()) {

Let's use a use statement.

Comment #8

blackra commented 15 May 2015 at 20:22

#7-1: The reason that the method is public is because there are other parts of the code that need this functionality for resolving other issues. Maybe it should be protected until those issues are addressed in case they don't happen until 8.1.x (minimum API changes).
Fabian, do you have any comments?

I think the rest of those comments seem sensible.

Comment #9

blackra commented 15 May 2015 at 21:47

Status	File	Size
new	2488730-9-check-for-apcu-version.patch	4.12 KB

Updated in line with comment #7.

I have also made the other helper member protected: protected static function checkForApcu()

Note that I used a protected static member cache rather than a local cache deliberately to allow for easier overrides.

Comment #10

blackra commented 15 May 2015 at 22:21

Issue summary:

View changes

Updated the issue summary.

Comment #11

fabianx commented 19 May 2015 at 14:43

Status:

Needs review

» Needs work

As much as I love it, there are other things in core calling function_exists('apc_fetch') and this should replace all of those ...

Comment #12

blackra commented 22 May 2015 at 00:42

Issue summary:

View changes

Status	File	Size
new	2488730-12-check-for-apcu-version.patch	5.49 KB

Updated to address #11

Comment #13

blackra commented 22 May 2015 at 00:43

Status:

Needs work

» Needs review

Comment #14

22 May 2015 at 00:44

Status:

Needs review

» Needs work

The last submitted patch, 12: 2488730-12-check-for-apcu-version.patch, failed testing.

Comment #15

blackra commented 22 May 2015 at 00:55

Status:

Needs work

» Needs review

Status	File	Size
new	2488730-14-check-for-apcu-version.patch	5.51 KB

Reroll of #12

Comment #16

fabianx commented 22 May 2015 at 01:32

Little nit:

+++ b/core/lib/Drupal/Component/Utility/Environment.php
@@ -42,4 +49,71 @@ public static function checkMemoryLimit($required, $memory_limit = NULL) {
+    $compatible_apc = false;
+
+    // Have we already cached the answer?
+    if (isset(Environment::$cache_check_for_compatible_apc_user_cache)) {
+      $compatible_apc = Environment::$cache_check_for_compatible_apc_user_cache;
+    }
+    else {

We can directly return here and spare the remainder.

$compatible_apc is not needed before the below.

Comment #17

blackra commented 7 June 2015 at 05:47

Status	File	Size
new	2488730-17-check-for-apcu-version.patch	5.44 KB

Little nit in #16 addressed plus a re-roll to deal with changes in \Drupal\Core\DrupalKernel

Comment #18

blackra commented 25 October 2015 at 17:58

Issue summary:	View changes
Status:	Needs review	» Closed (duplicate)

Originally, when this issue was encountered, Drupal 8 had a minimum PHP version of 5.4. This has since been raised to 5.5.9. It is believed that the faulty APCu was no longer being distributed by the time PHP 5.5.9 was released. Therefore, this issue is being marked as closed. The issue that forced the increase in PHP version was #2508231.

Comment #19

blackra commented 25 October 2015 at 18:01

APCu cache key corruption causes installation failures

Problem/Motivation

Observed Symptoms

Environment

Detail

Cause Hypothesis

Proposed Resolution

Comments

Comment #1

Comment #2

Comment #3

Comment #4

Comment #5

Comment #6

Comment #7

Comment #8

Comment #9

Comment #10

Comment #11

Comment #12

Comment #13

Comment #14

Comment #15

Comment #16

Comment #17

Comment #18

Comment #19

Related issues

Referenced by

News items

Our community

Documentation

Drupal code base

Governance of community