DX: Creating lazy services is too difficult/obscure/bespoke/brittle

Problem/Motivation

I was looking into creating a lazy service, since I knew this was possible in the Symfony DI component. There isn't much documentation on this in Drupal (see #2514582: Document lazy services and fix script doxygen) though it would be easy to think this works the same as in Symfony, which would be incorrect.

Lazy services were first introduced in #1973618: DIC: Lazy instantiation of service dependencies (ProxyManager for "proxy services"), in which the Symfony approach of a compiler pass was passed over in favor of a custom ProxyBuilder and ProxyDumper. The reasoning in the IS was:

This is the same approach as Symfony documents, except instead of using the Ocramius ProxyManager library that Symfony recommends, we use a custom lightweight implementation, which is faster (e.g., calls the proxied object's methods directly rather than via reflection) but not as complete (e.g., doesn't proxy property access, only method calls, and doesn't proxy magic methods). We think that's a worthwhile trade-off, since Drupal services don't use public properties, magic methods, etc.: the whole point of a service is to be defined by an interface.

This approach was later found to be incompatible with contrib modules, as the execution order of operations in building and dumping he DI container meant contrib interfaces were not discoverable. #2408371: Proxies of module interfaces don't work introduced the requirement to manually create the proxy service, using a script shipped with core.

There are a number of issues, such as #2532126: [meta] Follow-up for #2408371: Fix various issues with proxy services, #2539494: Provide a nice error message when trying to run the generate proxy class script on uninstalled/missing modules, the above-linked documentation todo, and #2513336: Consider checking checksum of interfaces during container building for proxy services which is probably most significant, having to do with the generated proxies going stale.

The vast majority of this work (and isuses) were done/logged 6+ years ago. From a DX perspective, I've personally concluded that the benefit of creating a lazy-instantiated service in Drupal is more trouble/danger than it's worth.

Steps to reproduce

Proposed resolution

Drupal's DI container differs from the upstream Symfony component in that it is serialized and dumped into the database. That introduces some significant challenges, but I'm optimistic that recent improvements in Symfony, paired with efforts in Drupal to come back closer to feature parity with upstream, might mean we can back out of the very Drupal-ized lazy service pattern. This would help site owners/developers, allow us to delete a lot of code from Drupal core, and make this less of a DX WTF.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet