Problem/Motivation

#2926309: Random fail due to APCu not being able to allocate memory found out that we'd disabled apcu_shared_prefix as a stop-gap for random test failures, fixed the cause of the test failures by disabling garbage collection, then left apcu_shared_prefix disabled resulting in random test failures again.

Re-enabling this brings down the number of apcu entries on the test infrastructure from 5M to 1M, and 300k using APCu file backend.

Opening this follow-up to discuss this issue further since 1M or 300k is still a lot, and traditional multi-site configurations, or sites with a /lot/ of config entities and translations could still run into some version of this issue.

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Comments

catch created an issue. See original summary.

andypost’s picture

andypost
he/him
Primary language Russian
commented

Also it may affect shared hostings

Maybe betrer to update test runners to manage prefixes?

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented

Just to note that purging the cache in test teardown will require an extra request to the webserver. The teardown method is being run via CLI and it has no access to the APC memory used by the webserver.

mpdonadio’s picture

mpdonadio
he/him
Primary language English
Location Philadelphia/PA/USA (UTC-5)
commented

Can it be in the __destruct() for the test base? Or the backend class?

class ApcuBackend {

  public function __destruct() {
    if (is_in_testbot()) {
       apcu_delete(new APCUIterator('/^' . preg_quote($this->binPrefix) . '/'));
    }
  }

}
alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented

@mpdonadio as per #3 that's not going to work. We need to trigger something on the web server. Doing something in the PHP cli process is not going to work and it can't happen on every request.

catch’s picture

catch
he/him
Primary language English
commented

Yes we'd need to have a route that checks we're in a test (possibly is conditionally added only when inside a test?), and flushes the apc prefix when requested.

Another thought:

The shared apcu prefix makes loads of sense for FileCache, but does it make any sense at all for config, can't see how it ever could with uuids. Should we look at making that more granular so that only shareable items get shared and then when we flush a prefix only install-specific items might get flushed?

Anonymous’s picture

Anonymous (not verified) commented

We do bit research (#148 - #172). TL;DR:

  • #3, #5: absolutely true)
  • #4, #6.1: my implementation of clearing by prefix via APCUIterator has led to a hangup (#150). And drupal_flush_all_caches() did not help too (#167)
  • While the best result showed the use apcu_store() with ttl (#171). Max number of entries: 68k (ttl 10s), 120k (ttl 200s).

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Related issues: +#2984232: APCu class loading does not automatically distinguish between sites in a multisite that has per-site code

@catch FYI: #2984232-10: APCu class loading does not automatically distinguish between sites in a multisite that has per-site code.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

catch’s picture

catch
he/him
Primary language English
commented
Priority: Major » Normal

This is less of an issue now we're on gitlab because we split tests between multiple test runners, compared to when all the tests ran on one big runner. However there could still be something going on here. Downgrading to normal though.