Discuss a better system for discovering and selecting normalizers

A few things I've thought about in some of my serializer work:

• I have more formats than encoders (e.g., more ways of structuring data for use in a JSON payload than ways to convert arrays to JSON). Having to create a "fuzzy-matching" JSON encoder and then a service for each format leveraging the same class seems like a cheat.
• I have formats that are not applicable to all data types, and there is no way explicitly declare that (e.g., REST UI, don't bother offering this format for REST plugins X, Y, or Z)
• Seems strange that high priority normalizers of irrelevant formats need to be checked. There's no mechanism to simply load all normalizers associated with format X.
• Drupal has extended the Serialization component above the level where some of the Symfony-documented serialization features are made available, such as cycle prevention.

Nice write up! This should be useful.

- The REST formats stuff, you mean you would like more granular control over what formats are allowed on specific routes/actions? Either way, seems slightly outside of the serializer itself.
- Yes, this is one of the things we've talked about before. Doing something that checks formats for normalizers. The trouble (or benefit) of the symfony serializer is that it relies on supportsNormalization() to check data and/or format. So it needs to iterate in order to find the first match. We could, if we didn't use Symfony anymore, could do something similar to our access checking system, where we have the notion of both a static and dynamic way to check if a normalizer applies based on format.
- What issues are you hitting around the loop prevention? I mean if we needed to, we could add the circular reference checks ourselves, like the Symfony normalizers do. Is that what you mean?

- The REST formats stuff, you mean you would like more granular control over what formats are allowed on specific routes/actions? Either way, seems slightly outside of the serializer itself.

For example, the description formats associated with Schemata will never be all that interesting with any of the content type resources unless I change the way Schemata interacts with REST. Why should inapplicable formats be offered in the REST UI for each resource? While this is outside the serializer, it goes to whether or not formats should have a more explicit relationship to the types of structures they transform. The answer might be "this is outside of scope, but maybe we should have a helper service that produces lists of viable formats for a given object to help build better UIs.

- What issues are you hitting around the loop prevention? I mean if we needed to, we could add the circular reference checks ourselves, like the Symfony normalizers do. Is that what you mean?

I have created a customized version of the hal_json format which recursively embeds referenced entities and certain cases of back-references. On top of that, I am doing some property exclusion and renaming to create a more approachable design for integration of these REST resources by non-Drupal developers. This has raised two problems:

1. Since I don't have access to more of the Symfony component, I had to do these kinds of implementations without the support. (Which in turn has made it more confusing when onboarding developers to serialization, telling them to ignore parts of the Symfony docs.)
2. As I realized only later, since I can't set cache tags from inside the serializer, my REST resources are in violating of the caching system. Recursively traversing the entities via a customized REST plugin seems like the only option to identify the entities in a location where I can add the cache tags; however, that seems pretty redundant so I've back-burnered that problem in hopes some better options emerge.

Didn't mean to raise the caching issues here as I think it's out of context, but I think it's helpful when thinking about the issues around when circular references are problematic.

Would it not make more sense for the Schemata module to not provide a REST resource plugin, precisely because it cannot ever offer >1 format? Then you can just ensure that your module provides one route, that cannot be modified via config.

The Acquia Content Hub module struggles with a similar need. I opened #2822201: Allow contrib modules to deploy REST resources by creating routes, removing the need for config entities in that special case for that use case. It'd be great to have two modules using this capability.

Schemata is going to drop rest plugins for routing, but its core architecture is built around serialization formats, one for each schema standard as applied to a "normal" format. So there is a "JSON Schema for HAL" module which is mostly made of normalizers.

I assemble the data needed to describe another resource then transform it into different serialized formats. Seems like a perfect fit.

Awesome :)

As an intermediate step, before we consider rolling our own serializer (which might actually be an ideal endgame considering cacheablility/bubbleable metadata etc..) I think we should try to implement a serializer instance PER FORMAT. We could instantiate each instance with the normalizers that are applicable only. This would make everyones lives much easier. We avoid the global normalizer rat race that we currently have. We could then have some kind of ChainSerializer that delegated to the correct serializer instance.

#11: AMEN.

In fact, @damiankloip and I discussed that very thing in IRC yesterday, referring to #2800873-59: Add XML GET REST test coverage, work around XML encoder quirks:

16:07:08 <damiankloip> WimLeers I think I see what you mean, you mean the not using the serializer to normalize the data
16:07:10 <damiankloip> just encode it
16:07:12 <damiankloip> ?
16:11:04 <damiankloip> The hal_json encoder only works afaik because it extends the json encoder and uses hal_json for the format
16:11:33 <damiankloip> I feel like I'm missing something really fundamental that you're saying - sorry!
16:37:21 <WimLeers> damiankloip: yep that's what I meant
16:37:31 <WimLeers> damiankloip: serializer = running normalizer then encoder
16:37:38 <WimLeers> damiankloip: deserializer = running decoder then denormalizer
16:37:50 <damiankloip> yep
16:37:54 <WimLeers> damiankloip: deserialize = string -> array -> object
16:37:59 <WimLeers> damiankloip: serialize = object -> array -> string
16:38:00 <WimLeers> that's all
16:38:41 <WimLeers> damiankloip: I guess Symfony fucked up its own abstraction/infrastructure
16:38:43 <WimLeers>   serializer.encoder.hal:
16:38:43 <WimLeers>     class: Drupal\hal\Encoder\JsonEncoder
16:38:43 <WimLeers>     tags:
16:38:43 <WimLeers>       - { name: encoder, priority: 10, format: hal_json }
16:38:56 <WimLeers> but \Drupal\hal\Encoder\JsonEncoder simply extends \Symfony\Component\Serializer\Encoder\JsonEncoder
16:39:03 <WimLeers> it only declares that it supports the "hal_json" format
16:39:11 <WimLeers> so it's not like it's a HAL+JSON specific encoder
16:39:19 <WimLeers> in fact, I'm cleaning up that very thing in …
16:39:32 <damiankloip> yeah, that's what I was saying above somewhere
16:39:41 <damiankloip> it just copies it to work for that custom format
16:39:42 <WimLeers> damiankloip: alright, good to hear we're on the same page
16:39:47 <damiankloip> I think we are :)
16:40:14 <damiankloip> WimLeers I am not the biggest fan of the serializer implementation, that's for sure
16:40:17 <WimLeers> hehe
16:40:19 <damiankloip> symfony that is
16:40:20 <WimLeers> damiankloip++
16:40:37 <WimLeers> damiankloip: the docs make it out to be better than it really is
16:40:46 <damiankloip> WimLeers totally!
16:40:54 <damiankloip> It sounds good
16:40:57 <damiankloip> until you start to use it
16:41:01 <WimLeers> haha exactly
16:41:01 <damiankloip> WimLeers I would like to consider that we set up a serializer per format or something
16:41:04 <WimLeers> that's my impression now too
16:41:10 <damiankloip> yeah :/
16:41:11 <WimLeers> yes
16:41:19 <damiankloip> but the ship sailed 'n' all that
16:41:23 <damiankloip> we'er stuck with it for now
16:41:23 <WimLeers> let formats ACTUALLY compose a normalizer and an encoder
16:41:29 <damiankloip> yes
16:41:31 <damiankloip> exactly
16:41:40 <WimLeers> damiankloip: I'm so glad you're +1 on that
16:41:56 <damiankloip> then some factory class can handle grabbing the right one
16:42:00 <WimLeers> damiankloip: I'd forever been wondering why the hell docs said "composable" but code said "cobbled together"
16:42:05 <damiankloip> I have wanted to do that for years now
16:42:13 <damiankloip> HAHA
16:42:13 <WimLeers> damiankloip: Symfony forced this on us
16:42:23 <WimLeers> damiankloip: I just thought for a while we got it wrong
16:42:38 <damiankloip> yeah, alas, it's just a royal pain in the ass
16:42:41 <WimLeers> damiankloip: I'm glad to have my understanding confirmed by you!
16:42:50 <damiankloip> :)
16:43:03 <WimLeers> damiankloip: and the priorities
16:43:09 <WimLeers> damiankloip: the normalizers and their priorities
16:43:21 <WimLeers> damiankloip: shall I fetch a nice fork to stab our eyeballs with?
16:43:43 <WimLeers> damiankloip: it's the whole asset weight, module weight, form element weight in Drupal, REPLICATED in Symfony
16:43:45 <damiankloip> please do, that would be much more pleasant than dealing with normalizer priorities
16:43:48 <WimLeers> just named differently
16:43:55 <damiankloip> yep

Or maybe even just some ChainNormalizer class could also solve the problem.

Hm, how do you envision that would work?

It could essentially group normalizers, so you would have a ChainNormalizer for hal, that would always run first if the format was hal (so it would first only check the format before passing on to the child normalizers contained within it), then fall back to default ones. Something like that. Haven't thought it through completely, just bouncing ideas.

That's an interesting direction for sure. I like it.

Something I may have mentioned above, but wanted to underscore: we have lots of ways to modify an existing format, but if you wanted to create a new format based on the normalizers for an existing one, things get dicey pretty fast. I've done this a couple different ways, both bad:

1. Created a trait to override the formats property or related support methods, then extend every normalizer I want to use into my format, using the trait as a fake base class. This sometimes causes a bad break in the magic of normalizers, which lets you end-run certain aspects of it's recursive nature by calling parent methods that get you out of the current data structure assumption.

2. Created a normalizer in which I did something to the effect of: $normalized = $this->serializer->normalize($entity, 'hal_json', $context); then hacked on top of the normalized result to correct any minor issues.

From a customization standpoint, I would love to be able to declare a format and have the format reference normalizers it should use. That way if I wanted my custom application/vnd.example.com.hal+json I can ditch a lot of awkward boilerplate and skip some procedural workarounds.

Yet another reason why we need this: #2841027: JSON API format should only be used on JSON API routes, but can currently also be used for REST resources.

This issue was mentioned in a Twitter discussion. In that discussion, dawehner mentioned a potential alternative: http://jmsyst.com/libs/serializer.

(See https://twitter.com/da_wehner/status/889970150836981760.)

I haven't had a chance to do a review of http://jmsyst.com/libs/serializer, but wanted to illustrate/underscore some use cases I've been thinking about with the Schemata project.

• How can gracefully detect if a requested serialization format is available?
• How can I list supported schema providers?
• How can I describe the schema formats

My current thinking is that this sort of Drupal-ly UI building is better served by creating a Schema Type plugin to serve as a metadata proxy for the collection of normalizers. Even if we could include this kind of information directly from encoders or normalizers, neither of those is 1:1 for the number of supported formats. It still seems awkward to create a plugin type largely for annotation collection.

Just want to throw the decorator pattern in the ring. It may not be helpful at all regarding this issue, but it's a pattern that's quite often forgotten but Symfony supports this pattern natively besides priorities for services.