Problem/Motivation

Every PHPUnit test method in core runs in its own PHP process (#[RunTestsInSeparateProcesses], enforced in BrowserTestBase and KernelTestBase). Each process compiles about 2,300 PHP files into a private opcache segment and throws it away when it exits. CLI processes cannot share opcache shared memory, so a CI job compiles Drupal thousands of times.

opcache.file_cache lets separate CLI processes share compiled scripts through a directory on disk. Only the first process compiles a file; later processes load it.

Proposed resolution

Enable opcache.file_cache for the PHP CLI only, in the CI test jobs:

mkdir -p /tmp/opcache-cli && chown www-data:www-data /tmp/opcache-cli
printf '\nopcache.file_cache=/tmp/opcache-cli\n' >> "$(php -r 'echo PHP_CONFIG_FILE_PATH;')/php-cli.ini"

in .gitlab-ci/scripts/server-setup.sh. Only php-cli.ini changes: Apache (mod_php) reads php.ini, so the web server is unchanged. A longer-term alternative is to set it in the DrupalCI images' php-cli.ini.

Measurements

Ratio = with cache / without cache, per test class, paired, with 95% confidence intervals.

Setup Kernel Functional
Dedicated host (Xeon E3-1245 v5), 3 interleaved rounds, 40 Kernel and 39 Functional classes, run-tests.sh concurrency 9 0.715 [0.699, 0.731]; wall 79.8 s to 58.3 s; job CPU -28% 0.870 [0.858, 0.883]; wall 104.4 s to 93.7 s; job CPU -14%
Second host, same hardware, some background load 0.729 [0.694, 0.764] 0.910 [0.888, 0.933]
GitLab CI, both arms in one pipeline, shard pairs on the same CPU model all 4 shards faster (0.49 to 0.83); no same-model pair class time 0.77 to 0.92 (about 0.87); job CPU 0.77 to 0.89

Pairing by CPU model matters: CI runners mix Xeon 8124M/8275CL/8375C/8488C and EPYC 7R13/7R32 nodes, and node placement alone moved Kernel timings by up to 25% between pipelines.

Tested and rejected: opcache.jit=tracing made tests 12% to 36% slower; a larger opcache.interned_strings_buffer (4 to 32 MB) made no difference; a tmpfs cache directory gave no measurable gain over the container filesystem.

Correctness and risks

  • The full test suite passed with the cache in GitLab CI. PHPStan produced byte-identical results with and without it.
  • validate_timestamps stays on, and opcache does not cache a file modified in the last 2 seconds (opcache.file_update_protection), so settings files and generated PHP that tests write are not served stale.
  • If the directory is missing or not writable, PHP aborts every CLI run ("opcache.file_cache must be a full path of an accessible directory"). The script must create it.
  • opcache creates its cache subdirectory with mode 0700 for the first user. If root compiles a PHP file first, www-data silently runs uncached. In the test jobs www-data is the first user: server-setup.sh runs no PHP as root, and the first PHP the jobs run is composer run-script as www-data.
  • About 70 to 225 MB of disk per job.

Remaining tasks

  • Merge request with the setup script change.
  • Confirm in a pipeline that the cache fills (file count in the job log) and compare job durations.

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3628411

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

andypost created an issue. See original summary.

andypost opened merge request !17339

andypost’s picture

andypost
he/him
Primary language Russian
commented
Status: Active » Needs review

andypost opened merge request !17363

longwave’s picture

longwave
he/him
Primary language English
Location UK
commented

@andypost as we discussed in Slack I think we should just configure this in php.ini in the CI images instead, then it makes all test runs faster for everyone.

andypost’s picture

andypost
he/him
Primary language Russian
commented

@longwave Agreed, the setting is moved into the 8.5/8.6 images. Pushed to drupalci_environments dev and cherry-picked to production

In median it shows 17% faster but kernel tests benefits the most

  • 7e8381d: PHP 8.5 apache and unit images
  • eeaf9b7: PHP 8.6 apache image

Two things came up while measuring the MR that the image has to handle as well:

  1. opcache.record_warnings=1 is required. Without it, a compile-time warning, such as a PHP 8.6 deprecation, is raised only by the first process that compiles the file. That process may be run-tests.sh rather than a test. With PHPUnit 12.5 --fail-on-deprecation on 8.5.11 and 8.6.0RC2, a warm cache without it reported OK. With it, the results match a run with no cache: each deprecation is reported once, and @/error_reporting() masks still apply.
  2. The first user to run PHP owns the cache. Opcache creates its subdirectory 0700 for that user. The image creates the directory 1777 and empties it after the root-run build steps (composer, drush). A job that runs PHP as root before the tests still ends up with a root-owned subdirectory, and the tests then silently run without the cache. Core's test jobs are fine, but PHPUnit Unit (Core) runs composer install as root first. A core pipeline on the new dev images confirmed it (997577): every test job got the cache as www-data, while Unit (Core) ended up with a root-owned cache holding only composer's files. Contrib's gitlab_templates has the same problem: it runs phpunit --version as root. So one line in core stays: a chown after composer.

longwave changed the visibility of the branch 3628411-ab-bench to hidden.

longwave’s picture

longwave
he/him
Primary language English
Location UK
commented
Status: Needs review » Reviewed & tested by the community

Let's do it. We can try to solve composer as root elsewhere if we want to, not sure it's worth doing though.

catch’s picture

catch
he/him
Primary language English
commented
Status: Reviewed & tested by the community » Needs work

The most recent job here is 4m19s which until two days ago would have equalled the record for fastest job. Very rare we get runs that fast so this is encouraging for a general improvement.

Do we need a gitlab_templates issue to add creating the directory there too?

Wanted to commit this but Drupal\Tests\views\FunctionalJavascript\Plugin\views\Handler\FieldTest is consistently failing on the MR. I can't reproduce it locally on either main or the MR branch.

andypost’s picture

andypost
he/him
Primary language Russian
commented

tested locally and still can't reproduce, the strange is that FieldTest, the repeated failure is testModalDialogTitle at line 115 expected "Content", actual empty string.

The only difference is that the test passed on shard 2 but last runs were in shard 3, will dig deeper

PS: https://git.drupalcode.org/project/gitlab_templates/-/work_items/3572454

longwave’s picture

longwave
he/him
Primary language English
Location UK
commented

I assume this is just making a race condition that was always there much easier to hit now things are faster. But we should try to solve it or it will just become another annoying random fail...

andypost’s picture

andypost
he/him
Primary language Russian
commented
Status: Needs work » Needs review

Debugged and fixed test - as it running faster now the browser have less time to position element so waitForElementVisible() solved the issue

longwave’s picture

longwave
he/him
Primary language English
Location UK
commented
Status: Needs review » Reviewed & tested by the community

I came to the same conclusion, that we are racing the 20ms debounce delay and that waiting for the element to actually become visible is the correct fix.

@catch the test already calls assertWaitOnAjaxRequest so I think that part should be fine?

  • catch committed 6ee54f81 on 12.0.x 
    task: #3628411 [ci] Share compiled scripts between PHPUnit processes...

  • catch committed 43689224 on main 
    task: #3628411 [ci] Share compiled scripts between PHPUnit processes...
catch’s picture

catch
he/him
Primary language English
commented
Version: main » 12.0.x-dev
Status: Reviewed & tested by the community » Fixed

@longwave ::assertWaitOnAjaxRequest() is the old buggy method that is unfortunately not deprecated yet, ::assertExpectedAjaxRequest() is the new robust method. But let's let the issue changing all occurrences address that since the race condition is not the fault of this issue at all - it's just exposing it.

Committed/pushed to main and 12.0.x, thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

andypost’s picture

andypost
he/him
Primary language Russian
commented
Related issues: +#3401619: Deprecate ::assertWaitOnAjaxRequest() in favour of ::assertExpectedAjaxRequest()

Thank you, gonna explore the related issue