[META] Normalization System: clean up/speed up/provide schema

Over the past year, we've added comprehensive test coverage and ensured that JSON:API was both secure and cacheable (by auditing and correcting cache contexts/tags)

In this process, we learned that our normalizers were incompatible with the Symfony normalizer interface and we "cordoned off" the normalization system with the eventual intent of refactoring it and adding support for schema generation/validation.

This issue is meant to track and coordinate that process. Below, I've written out in broad strokes the steps, in order, that I think we can take to end up in a world where our normalizers aren't a black box to outsiders, where they're faster, where they're schema-complete and where there's a clear path forward for bringing Drupal core's normalizers into that fast/schema-complete world too (with minimal disruption and no BC-breaks).

1. I'd like to ensure the our many normalizer value objects really are just value objects. Over time, we've added many more value objects and learned that some of their inheritance hierarchies make them unwieldy (HttpExceptionNormalizerValue extends FieldNormalizerValue, for example). Some of them implement a hefty bit of specialized normalization logic for what should really pure value object. I believe this first step will result in fewer value objects. I think we can consolidate many of them into a standard NormalizedValue object which simply takes an array and a CacheableMetadata object.
   • #3022584: Consolidate and simplify NormalizerValue objects: introduce CacheableNormalization
   • #3024896: Remove Relationship and RelationshipItem classes

2. I'd like to reduce the number of (or at least confirm the number is necessary) normalizer's that we actually have. I believe issue like #3015438: Wrap entity objects in a ResourceObject which carries a ResourceType can start to reduce the number of them. I think the Relationship and RelationshipItem classes (and their normalizers) can probably be factored away if we take advantage of the new value objects we have elsewhere (like EntityCollection and ResourceIdentifier(Interface)).
   • #3015438: Wrap entity objects in a ResourceObject which carries a ResourceType

3. I'd like to come up with a pattern to extend the now cleaned up normalizers with a caching mechanism. Right now, I'm imagining a service tag that identifies the normalizer as "deterministic", meaning that it can be cached by its $supportedInterfaceOrClass value. (Done in #3252872: Use CacheableSupportsMethodInterface for performance improvement in normalizers) Following up on that, I think we could use reflection to identify TypedData normalizers (IOW, non-JSON:API normalizers) that can also be cached and use that knowledge to alter their service definitions in order to add the "deterministic" tag.
   • #3014283: Use ResourceTypeFields to statically determine the field normalizer
   • #3031214: Introduce "deterministic" normalizers
4. At this point in the process, I think it would be safe to then begin adding the "deterministic" tag to core normalizers where possible and altering the Drupal core serializer service to implement cacheing based on this tag as well. All normalizers probably won't be 100% cacheable, but I think we could get to that state for most of them. - No longer a need for deterministic tag.
5. After the "deterministic" flag and concurrently with core's adoption of it, I'd like to add two new interfaces called SchematicNormalizerInterface and SchematicDenormalizerInterface. These interfaces would have a get(De)NormalizationSchema method on them. JSON:API could enforce this interface for its own normalizers and begin adding it to core normalizers as well. With the addition of that interface, we could then validate input and output of those normalizers against their declared schema. The challenge here is that normalizers for things like EntityInterface can't really provide a useful schema without knowledge of the entity type/bundle This is where @e0ipso's ResourceFieldInterface et al. come into play by allowing the ResourceType to know about its field types. I think this could work and perhaps the new schema interface will have to take a queue from (or import) @e0ipso's shaper library with its concept of normalization "context" (this is different from Symfony's idea of context, I believe).
   • #2994473: [META] JSON API's normalizers support schema tracking, to guarantee comprehensive schema
6. Paired with the "deterministic" flag and the schema interfaces, we'd then have enough knowledge to generate resource type-level schema's with some certainty. Contrib and core would have a BC-compliant way of opting in to this system by adding the "deterministic" tag and implementing the additional interface(s).
7. With this concept proven, I think we could then require new normalizers to have this tag and interface on them and move those tags/interfaces into Drupal core. When normalizers are detected without either of them, we could throw deprecation notices and then require that they be implemented in Drupal 11.
8. After (or maybe even before) these deprecations, I think that we could have a schema generation module (probably schemata) that with a config/settings.php setting, we could require these tags/interfaces and if the normalizer does not have both, it could simply remove the service from the container. If the settings is not enabled, then the schema it would output would fall back to using any for non-deterministic scenarios. that outputs a schema with schemas, if known, or any or some other safe default in other cases.

Obviously, the above is subject to lots of input and suggestions. I've been thinking about all this for a while but have never actually put it all down in one place.