When working on #3583040: Cache the container definition in APCu, I learned that the cached container definition is 500KB+ of serialized data. I was like: "Why is it so large?". Because loading the cached container is in the critical performance path, I decided to see what is actually in it.

One of the things I noticed is that it contains .hook_data (~ 45KB) and .theme_hook_data (~10 KB). I believe these are temporary build-time data. They should not appear in the final cached container. In total this is a bit more than 10% of the container.

Symfony provides a RemoveBuildParametersPass for this purpose: parameters prefixed with . are a Symfony convention for build-only data. Drupal's compilation pipeline never registers this pass, so the data leaks into the cached container definition.

The solution is to have HookCollectorKeyValueWritePass remove the temporary parameter. I've not benchmarked this but it should result in a small performance improvement for all requests.

Related issues

Issue fork drupal-3583125

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

dries created an issue. See original summary.

dries’s picture

Issue summary: View changes
dries’s picture

I think we can just register Symfony's built-in RemoveBuildParametersPass in CoreServiceProvider.

I measured this using Umami and the container cache drops from 549 KB to 482 KB (~10% reduction).

Loading and unserializing the container takes a good amount of time so this might help performance.

longwave’s picture

#3571858: Optimize OptimizedPhpArrayDumper further also attempts to reduce the container size, both of these together would be even better.

longwave’s picture

Maybe we should add a performance test for container size, so we get notified by a test failure if the size increases by a certain amount in any MR.

berdir’s picture

Yes, I think we explicitly used dot-prefixed assuming that they would get removed. FWIW, these structures are maybe 10-20% of the size that they had in 11.2 but it's of course still useful to remove every bit we don't need.

nicxvan’s picture

Yeah I guess we should have verified that the symfony docs aligned with drupal's implementation. We explicitly used .hook so it would be removed.

nicxvan’s picture

I guess the main question is do we want to add a compiler pass or manually delete the two we have.

The obvious benefit of the new pass is we can use . Parameters more freely and not worry about cleanup.

Alternatively we could just require manual cleanup when we use them and have a canary test that looks for .prefix parameters in the compiled container.

dries’s picture

Status: Active » Needs review

@nicxvan: I agree, that is the main question. I actually implemented both approaches and ended up choosing the compiler pass because it was only one line of code and felt like the "correct way" to go. Either approach works for me though. Who makes that call?

longwave’s picture

Status: Needs review » Needs work

Adding the Symfony pass seems fine to me, it's only one line and to do this ourselves would basically mean copying the Symfony code. I'm also looking at optimising the pass configuration over in #3581926: Filter out unused compiler passes.

We could do with some kind of test here to prevent a regression.

nicxvan’s picture

Who makes that call?

If there is a disagreement it would be the framework managers.

But this is build time and would prevent needing to handle this is each time we use more dot prefixed parameters. I agree now that the Compiler pass is better.

However, I think we should use PassConfig::TYPE_REMOVE instead of a hard coded priority though.

And we do need the regression test.

dries’s picture

Thanks for the review @nicxvan. We're already using PassConfig::TYPE_REMOVE. See: this commit. I just added a regression test. 👍

nicxvan’s picture

Ah yes, I was talking about the , -2048 that you removed.

I think this is ready, there seems to be a random failure if that is rerun and green I think we can mark this RTBC.

I did pull this locally and ran the test only job and it failed as expected:

There was 1 failure:

1) Drupal\KernelTests\Core\DrupalKernel\DrupalKernelTest::testBuildParametersRemoved
Dot-prefixed build parameters should not be in the compiled container, but found: .hook_data, .theme_hook_data
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
-Array &0 []
+Array &0 [
+    27 => '.hook_data',
+    28 => '.theme_hook_data',
+]
dries’s picture

Status: Needs work » Reviewed & tested by the community
godotislate’s picture

I was curious why RemoveBuildParametersPass wasn't running already, because it's part of the default PassConfig:

      $this->afterRemovingPasses = [
            0 => [
                new ResolveHotPathPass(),
                new ResolveNoPreloadPass(),
                new AliasDeprecatedPublicServicesPass(),
            ],
            // Let build parameters be available as late as possible
            // Don't remove array parameters since ResolveParameterPlaceHoldersPass doesn't resolve them
            -2048 => [new RemoveBuildParametersPass(true)],
        ];

Then I noticed that true is passed into the constructor, which per the comment means that parameters with array values aren't removed. Not sure why ResolveParameterPlaceHoldersPass is set not to resolve array parameters, though or why then dot prefixed array values shouldn't be removed.

The first false passed into ResolveParameterPlaceHoldersPass disables array resolving.

 $this->optimizationPasses = [[
            new AutoAliasServicePass(),
            new ValidateEnvPlaceholdersPass(),
            new ResolveDecoratorStackPass(),
            new ResolveAutowireInlineAttributesPass(),
            new ResolveChildDefinitionsPass(),
            new RegisterServiceSubscribersPass(),
            new ResolveParameterPlaceHoldersPass(false, false),
dries’s picture

Great find! Interesting that Symfony's default RemoveBuildParametersPass() preserves arrays. Looks like that was introduced in https://github.com/symfony/symfony/pull/62843. That probably impacted us. Good we have a test now.

dries’s picture

Status: Reviewed & tested by the community » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

longwave’s picture

Status: Fixed » Reviewed & tested by the community

This doesn't seem to be in the repo, moving back to RTBC for now.

longwave’s picture

Version: main » 11.x-dev
Status: Reviewed & tested by the community » Fixed

Committed and pushed 007adf94b0f to main and 7ff3e0390fe to 11.x. Thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

  • longwave committed 7ff3e039 on 11.x
    perf: #3583125 Strip dot-prefixed build parameters from cached container...

  • longwave committed 007adf94 on main
    perf: #3583125 Strip dot-prefixed build parameters from cached container...