Problem/Motivation

There is some dead code in OptimizedPhpArrayDumper that we could optimize away to make dumped containers a bit smaller.

We might also be able to optimize the structure a bit more, again to reduce size of the dumped container.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#6 ContainerDumpSizeTest.php_.txt8.26 KBlongwave

Issue fork drupal-3571858

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

longwave created an issue. See original summary.

longwave’s picture

Baseline serialized container: 285234 bytes
After these changes: 272228 bytes
Saving: 13006 bytes (4.6%)

longwave’s picture

We can also remove the default invalidBehavior case to save even more:

After second round of changes: 241675 bytes
Saving: 43559 bytes (15%)

longwave’s picture

Status: Active » Needs review

We can also remove the collection wrapper, and the raw type is unused too.

After third round of changes: 206691 bytes
Saving: 78543 bytes (27.5%)

longwave’s picture

Issue tags: +Performance
StatusFileSize
new8.26 KB

Final pass at this, heavily assisted by Claude Code.

This MR optimizes the machine-readable container format by encoding service references as bare strings instead of stdClass objects. Service references are the most common argument type (~53% of all arguments across core), so this reduces serialized size, object count, and resolution overhead.

In the optimized format, a bare string in an argument list is now a service ID with EXCEPTION_ON_INVALID_REFERENCE behaviour (this covers 99.6% of service references). This is the fast path in resolveServicesAndParameters().

Since bare strings now mean service IDs, literal string values are wrapped in a "raw" type object; these are much rarer than service references.

"Collections" are now always just plain arrays, and the container no longer needs to unwrap collection objects.

I also wrote a test that outputs various statistics and benchmark timings from a fully-loaded container with all core modules enabled; that code is attached to this comment.

  ┌────────────────────┬─────────────┬─────────────┬────────┐
  │       Metric       │ origin/main │ this branch │ change │
  ├────────────────────┼─────────────┼─────────────┼────────┤
  │ Total size         │ 757.0 KB    │ 502.0 KB    │ -34%   │
  ├────────────────────┼─────────────┼─────────────┼────────┤
  │ Services size      │ 591.6 KB    │ 336.8 KB    │ -43%   │
  ├────────────────────┼─────────────┼─────────────┼────────┤
  │ stdClass objects   │ 4,267       │ 1,347       │ -68%   │
  ├────────────────────┼─────────────┼─────────────┼────────┤
  │ Unserialize ALL    │ 0.94 ms     │ 0.50 ms     │ -47%   │
  ├────────────────────┼─────────────┼─────────────┼────────┤
  │ Unserialize top 18 │ 0.24 ms     │ 0.21 ms     │ -13%   │
  ├────────────────────┼─────────────┼─────────────┼────────┤
  │ get() ALL          │ 10.49 ms    │ 8.17 ms     │ -22%   │
  ├────────────────────┼─────────────┼─────────────┼────────┤
  │ get() top 18       │ 1.33 ms     │ 1.17 ms     │ -12%   │
  └────────────────────┴─────────────┴─────────────┴────────┘
longwave’s picture

Status: Needs review » Needs work

Had an idea that might make this even smaller with no CPU overhead: map service names to integers and use the integer values for argument references.

longwave’s picture

Status: Needs work » Needs review

Let's leave that for now, can be done in a followup

dries’s picture

I've been poking at this and want to float an idea.

The current approach makes bare strings mean "service reference" and wraps literal strings in a {type: 'raw'} stdClass. What if we went the other way: prefix the references instead? So '@module_handler' is a service reference, '%cache_bins' is a parameter, and 'hello' is just a literal string, no wrapper needed.

Your approach eliminates stdClass wrappers on service references, but every literal string argument becomes a new stdClass object. So we trade one set of allocations for another. I believe the prefix approach could eliminate both.

dries’s picture

I prototyped the prefix idea from #9 with the help of Claude Code and benchmarked it against longwave's latest MR. This builds on the great work longwave already did.

First, let me illustrate the encoding difference as it helps everyone understand it. Let's take module_handler as an example. This is a typical service with five arguments: one literal string, one parameter reference, and three service references.

Drupal main

Its dumped form looks like this (596 bytes):

'module_handler' => [
  'class' => 'Drupal\Core\Extension\ModuleHandler',
  'arguments' => stdClass {
    type: 'collection',
    value: [
      '/var/www/html',
      stdClass { type: 'parameter', name: 'container.modules' },
      stdClass { type: 'service', id: 'keyvalue', invalidBehavior: 1 },
      stdClass { type: 'service', id: 'callable_resolver', invalidBehavior: 1 },
      stdClass { type: 'service', id: 'cache.bootstrap', invalidBehavior: 1 },
    ],
  },
  'arguments_count' => 5,
]

Five stdClass allocations per container build for one service: one collection wrapper, one parameter wrapper, and one per service reference.

Longwave's latest MR

Longwave's latest MR flattens service references to bare strings and drops the collection wrapper (321 bytes):

'module_handler' => [
  'class' => 'Drupal\Core\Extension\ModuleHandler',
  'arguments' => [
    stdClass { type: 'raw', value: '/var/www/html' },
    stdClass { type: 'parameter', name: 'container.modules' },
    'keyvalue',
    'callable_resolver',
    'cache.bootstrap',
  ],
]

Two stdClass allocations remain: one wrapping the literal string /var/www/html (because bare strings now mean "service reference"), and one for the parameter reference.

Dries' alternative encoding

The prefix variant eliminates both of the remaining wrappers (216 bytes):

'module_handler' => [
  'class' => 'Drupal\Core\Extension\ModuleHandler',
  'arguments' => [
    '/var/www/html',
    '%container.modules',
    '@keyvalue',
    '@callable_resolver',
    '@cache.bootstrap',
  ],
]

Zero stdClass allocations. The literal string passes through unchanged with no prefix. The parameter reference uses a % prefix. Service references use an @ prefix. Every argument stays a bare scalar, with no heap allocation per reference.

Benchmark results

Size aside, the main advantage of my approach is that it's faster, which is ultimately what we're trying to optimize for. Across a full Drupal install with all core modules enabled, prefix encoding eliminates over a thousand additional stdClass allocations that longwave's MR still makes.

Using ContainerDumpSizeTest.php from #6, with all 72 core modules enabled:

┌────────────────────┬─────────────┬─────────────┬─────────────┬────────────────┬───────────────────┐
│       Metric       │ origin/main │  longwave   │   dries     │ dries vs main  │ dries vs longwave │
├────────────────────┼─────────────┼─────────────┼─────────────┼────────────────┼───────────────────┤
│ Total size         │ 726.2 KB    │ 475.7 KB    │ 405.0 KB    │ -44%           │ -15%              │
│ Services size      │ 577.5 KB    │ 327.2 KB    │ 256.5 KB    │ -56%           │ -22%              │
│ stdClass objects   │ 4,184       │ 1,298       │ 108         │ -97%           │ -92%              │
│ Unserialize ALL    │ 0.93 ms     │ 0.52 ms     │ 0.35 ms     │ -62%           │ -33%              │
│ Unserialize top 17 │ 0.26 ms     │ 0.25 ms     │ 0.11 ms     │ -58%           │ -56%              │
│ get() ALL          │ 7.00 ms     │ 4.96 ms     │ 4.54 ms     │ -35%           │ -8%               │
│ get() top 17       │ 0.92 ms     │ 0.96 ms     │ 0.71 ms     │ -23%           │ -26%              │
└────────────────────┴─────────────┴─────────────┴─────────────┴────────────────┴───────────────────┘
godotislate’s picture

Only took a quick look at the prefix solution in https://git.drupalcode.org/project/drupal/-/merge_requests/15349, and it seems like it accounts for escaping raw strings starting with @ and & with @@ and &&, but not %? Seems like it makes sense to use the same double escaping of %% as https://symfony.com/doc/current/configuration.html#configuration-parameters?

Granted these are likely edge cases.

dries’s picture

@godislate: We don't use @@/&& double-prefix escaping. For all three characters (@, &, %), literal strings that would collide with prefix encoding are wrapped in a stdClass with type => 'raw', which the resolver treats as literal values. For %, that is done by the str_contains block around line 380.

godotislate’s picture

@dries: Oh, OK. I was just going off the new testPrefixedLiteralStringEscape method in OptimizedPhpArrayDumperTest, which has test cases for string literals starting with @ and &, but not %.

longwave’s picture

Status: Needs review » Needs work

Added some questions and comments. Posting from my phone, hoping to review/test this from a computer later this week.

longwave’s picture

Alternative proposal in #3583505: Use Symfony PhpDumper instead of a serialized array container structure where we drop this array-based format entirely and swap to Symfony's PhpDumper. This was previously swapped the other way in 2015, maybe we can revisit it now.