Problem/Motivation
Several (most?) of JSON API's normalizers return objects as output, in direct violation of the Symfony NormalizerInterface that they all, ultimately, implement. This means that they are not playing nicely with other normalizers that do conform to the interface -- in fact, this broke Metatag at one point, and maybe still does.
Consider this piece of JSON API's EntityNormalizer:
$output = new EntityNormalizerValue($normalizer_values, $context, $entity, $link_context);
// Add the entity level cacheability metadata.
$output->addCacheableDependency($entity);
$output->addCacheableDependency($output);
// Add the field level cacheability metadata.
array_walk($normalizer_values, function ($normalizer_value) {
if ($normalizer_value instanceof RefinableCacheableDependencyInterface) {
$normalizer_value->addCacheableDependency($normalizer_value);
}
});
return $output;
NormalizerInterface says that $output must be an array or a scalar. This breaks that promise, which means that JSON API is essentially not interoperable.
Proposed resolution
Honestly, I don't know. There's probably a good reason why JSON API did this, but I guess the best resolution would be to find a way to refactor this, if necessary, so that it plays nicer with the serialization subsystem, and with other modules that implement normalizers.
Remaining tasks
No idea. TBD.
User interface changes
TBD.
API changes
TBD.
Data model changes
TBD. Hopefully none!
| Comment | File | Size | Author |
|---|---|---|---|
| #44 | 2923779--normalizers-refactor--44.patch | 20.18 KB | e0ipso |
| #42 | 2923779--normalizers-refactor--42.patch | 20.16 KB | e0ipso |
| #42 | 2923779--interdiff--29-42.txt | 6.65 KB | e0ipso |
| #29 | interdiff-2923779-24-29.txt | 1.87 KB | gabesullice |
| #29 | 2923779-29.patch | 22.67 KB | gabesullice |
Comments
Comment #2
phenaproximaComment #3
wim leersComment #4
e0ipsoClosing as duplicate of #2860350: Document why JSON API only supports @DataType-level normalizers.
You can read there why external normalizers that are not JSON API specific should not be supported by JSON API. The exception is the normalizers at the typed data level, which don't have the conditions described in this issue.
The current behavior is to explicitly ignore normalizers that would engage in the behavior described in the IS.
Comment #5
wim leersBut @phenaproxima was pointing out not that JSON API normalizers cannot be reused, he was pointing out that JSON API normalizers returning value objects violates
Comment #6
e0ipsoCan you provide a specific example of what you mean? I don't see a practical example where that would be a problem. You have to explicitly use
api_jsonformat. When you do, you need to deal with the format of the API calls, as with every other API.AFAIK docblocks are not part of an interface. In any case, returning
['the_thing' => $an_object]instead of$an_objectwould comply. However that will add only churn.A viable solution could be to change the internal normalizers to implement a custom interface instead of
NormalizerInterface. I think the best course of action is to detach JSON API more from Serialization, since it only adds confusion and false expectations.People tend to think cross-format re-use, which is not possible. See #2860350: Document why JSON API only supports @DataType-level normalizers for a longer explanation of why that is not possible.
Comment #7
wim leersI was merely replying to you in #4. I wrote . i.e. that's precisely not the point.
They are, because PHP 5 doesn't support return type hints.
Interesting idea!
Comment #8
wim leers@e0ipso at #2931785-15: The path for JSON API to core:
@Wim Leers at #2931785-16: The path for JSON API to core:
Comment #9
e0ipsoHmm. That's one of the reasons I'd like to get a first assessment of the core committers.
On the particular subject I don't think that there is an interface violation. PHP fails on interface violations. I am not convinced by the comment in #7:
More importantly, if we were to address this we can either:
… all this because of a
@return arraycode comment.Comment #10
wim leersVery interesting comment, @e0ipso, I'll need to think some more about that :) 👍
Do others already have thoughts about #9? @phenaproxima, @gabesullice?
Comment #11
gabesulliceI agree that we should not base our decision on speculation. Better to get a final answer.
This is a "letter vs spirit of the law" problem. @e0ipso is right, the letter of the law (PHP) doesn't care about return types. Such is PHP. @Wim Leers is right that the docblocks extend the interface beyond the limitation of of old PHP versions.
My opinion is that we are certainly in violation the spirit of the law here. I think we have all written code would be considered unsafe if not for a docblock's return type. Consider:
As @phenaproxima already pointed out, we're definitely violating a social contract, that's why we "broke Metatag".
So, in my eyes, this is a bug which needs to be fixed.
Since I see it as a bug, then we have a few choices about what to do about this bug:
So...
1 or 2 means we need to refactor. So, do we do that refactoring in contrib or do we do it in core? If a maintainer says this needs to happen before we go into core, then our decision is made for us. If not, we need to weigh a 6 month delay against the burden of refactoring this in core vs. contrib. My preference would be to do this in contrib.
3 seems pretty unacceptable. I could only see doing this as an option if we provided some kind of shim between our custom stuff and the serialization API.
Please forgive my ignorance here. I sincerely am not clear where the BC break is.
Comment #12
e0ipsoThat's not correct. Metatag was wrongly assuming they can provide normalizers for fields for a "generic format" and expect that to be valid in all formats. As we have now clarified (thanks to you) that is not possible. This problem is different.
I tend to agree. However, there is an important fact here: PHP does not support union of types. "Scalar or array" is not a thing in programmable PHP interfaces. Just like you cannot typehint that into a return value, hence the upstream lib will never add them in PHP7. Even if it was possible, altering that symfony interface is a BC break (and that is because a PHPdoc does not enforce & therefore does not guarantee).
Let's finish this discussion later in video :-)
Comment #13
phenaproximaTwo possible (reasonably) quick ways to work around this without major refactoring occurred to me. Neither way is perfect, but they're worth mentioning.
is_array()checks, which could be quite a gotcha if it's not documented anywhere.Comment #14
wim leers#13:
is_array()checks.serialization's, becauseserialization's are generic and will override anything else. Yes, the more specific normalizers are ignored if a generic normalizer is found. Yes, the Symfony Serializer component is that naïve. See #2575761: Discuss a better system for discovering and selecting normalizers.@e0ipso, @gabe.sullice and I just had a call, we'll summarize it soon.
Comment #15
gabesulliceSummary of videochat discussion:
@e0ipso started off with some history.
In short: he originally tried to use the normalizers “as intended”, but ended up with complex logic tracking state in parallel to the normalization process: state to track both cacheability metadata associated with each normalized item in the tree (
entity -> fields -> properties) *and* JSON API-specific metadata to ensure the right data ends up in the right places in the right place of the document structure: http://jsonapi.org/format/#document-structure.This is why he ended up returning value objects from each normalizer instead, and ending up with a tree of value objects, each containing their metadata (cacheability + JSON API), which can then be “rasterized”). See:
\Drupal\jsonapi\Normalizer\Value\JsonApiDocumentTopLevelNormalizerValue::rasterizeValue(),\Drupal\jsonapi\Normalizer\Value\JsonApiDocumentTopLevelNormalizerValue::rasterizeIncludes(), etc.)@e0ipso started the discussion with 3 key observations:
@return scalar|arraylimitation imposed by Symfony’sNormalizerInterfacetypehint, then we will again have to deal with the complex state tracking mentioned above, and attempted before.scalar|array)are unsupported. See https://stackoverflow.com/questions/36873567/php-return-type-hinting-obj... and https://wiki.php.net/rfc/union_types.Observation by @wim.leers Leers not mentioned in the call: "If Symfony really wanted to enforce this, they could’ve added anif-test orassert()call that would be guaranteeing ascalar|arrayreturn value."serializerservice. Then it wouldn’t violate the interface anymore.@wim.leers Leers explained that his key concern was not per se the strict compliance with an interface, but rather A) the impact on the API-First ecosystem in Drupal, B) whether core committers would accept this into core (extremely unlikely). We either need:
@wim.leers Leers piggy-backed off @e0ipso’s third observation, and proposed to essentially stop using the Symfony
serializerservice for JSON API’s “document top”-level, entity-level and field-level normalizers, and hopefully simplify that code thanks to no longer needing to conform to that structure. Only the field normalizer would then still call out to the “normal” serializer, and would hence pick up the@DataType-level normalizers. This would then better comply with the existing docs injsonapi.api.phptoo!@e0ipso observed, and we all agreed with him, that a good first step would be to at least “cordon off” the existing JSON API normalizers, so that they’re not available to contrib/custom code, by making them use a separate service tag. All of the code is already
@internal. If that tag is then also clearly for internal use only, then we can:NormalizerInterface!Therefore, the plan, which I'll be implementing:
Comment #16
e0ipsojsonapi_normalizer_do_not_use_removal_imminentnow that's a declaration of intentions! hahahaThis summary is a perfect recollection of the call. Many thanks @gabesullice!
Comment #17
wim leers😀
Comment #18
gabesulliceI can't take credit for the excellent summary, that was written by @Wim Leers. BUT, my sole addition was the tag name :) so there's that!
Comment #19
gabesulliceI tried to do this with as tiny a scalpel as I could. All of our normalizers are now completely segregated from the rest of Drupal's serialization system.
I was not able to remove
NormalizerInterfacefrom our normalizers or else I would have to rewrite a lot more of Symfony's code (which does type assertions on their interface). But I think we're sufficiently "cordoned off" that I'm okay with the interface violation for the time being, @Wim Leers, would you agree?Comment #21
phenaproximaNot sure how much my opinion counts here, but I totally agree. The new tag is quite specific and well separated from the rest of the normalizers. In my opinion, this is an excellent workaround.
Comment #22
e0ipsoVery. For years there was only my opinion and it was fine, then Wim got more involved and things were better, then Gabe jumped back in and things are even better.
The more opinions the better solutions.
Comment #23
gabesulliceIt absolutely does! I should have called you out specifically. I called @Wim Leers out because it was his suggestion to "4. remove “implements NormalizerInterface” from jsonapi_normalizer_do_not_use_removal_imminent services"
I didn't realize how the format thing worked. This should fix the test failure and make the patch smaller :)
Comment #24
gabesulliceFixing the missing newline...
Comment #25
wim leersLet's make this a private service.
Let's document why we need to specify these priorities. Will make future refactoring easier.
This will stop working when this service is marked private. It'll need to be injected.
<3
Oh, CLEVER! This means even fewer changes. Otherwise, we'd need to update JSON API's field normalizers to specifically use core's
serializerservice rather thanjsonapi's.This is a great first step :)
Supernit: Let's add a trailing\n:)Comment #26
wim leersNW for #25 points 1+2.
Comment #27
e0ipsoNow that we have an agreement and we have the ball rolling I'm increasing the priority on this.
I really like where we're headed.
Comment #28
wim leersMe too! :)
Comment #29
gabesullice#25.2: I commented on all the priorities and found one that was not necessary.
#25.1/3: I was not able to mark the service as private. I don't think this is possible because controllers are instantiated at runtime using the
static
create(ContainerInterface $container)method. Therefore, one must call$container->get('private_service');, which is not allowed.Let's make a follow-up to mark it as private once we have #2883086: [PP-1] Port RequestHandler + ResourceResponseSubscriber improvements from REST module to JSON API in, which might help find a cleaner way to do injections.
Comment #30
gabesulliceFollow-up created: #2935370: Mark the JSON API serializer, normalizer and encoder services as private
Comment #31
wim leersRE #25.1/3: d'oh you're right :(
I can only find one potential problem:
Instead of getting a specific normalizer service, this is changing that to use the JSON API serializer service. Maybe we should do that, but isn't that an out-of-scope change?
Comment #32
gabesulliceNot really. That's in a test class and it ensures that the normalizer is correctly implementing
supports(De)Normalization.Comment #33
wim leersAh, that makes sense!
Comment #34
e0ipsoI'm in the process of reviewing the code. However I'm missing a clear release change note for this issue since it may cause BC breaks on people writing their own custom JSON API-specific normalizers. We agree it's an edge case that's easy to solve (change your service tag) but we need to document that in the release notes.
FWIW whenever this is released it will go as the only commit in the release. This changes architecture and it should be easy to roll back / spot.
Needs work not for the code, but for the missing release notes.
Comment #35
gabesulliceComment #36
wim leersShouldn't one already have been using
\Drupal::service('jsonapi.entity.to_jsonapi')->serialize($entity)for that anyway?Everything else looks fine.
Comment #37
gabesullice@Wim Leers, updated my original comment.
Comment #38
e0ipsoPutting it back to RTBC as per #33. I'll add my review and commit soon.
Comment #39
gabesulliceCR created: https://www.drupal.org/node/2935900
Comment #40
wim leers🎉
Comment #41
e0ipsoSorry it took longer than expected, but the little one was sick this week.
Do we really need this? I get this will go away soon, but it sounds like we could simplify.
These (and others) seem like a regression. Why introduce a level of indirection to find
serializer.normalizer.jsonapi_document_toplevel.jsonapiinstead of calling it direcly?It also feels out of scope of the current change.
Comment #42
e0ipsoI didn't run tests locally, so this will surely fail.
Comment #44
e0ipsoRe-roll.
Comment #45
e0ipsoComment #46
e0ipsoHa! They passed after all.
Comment #47
e0ipsoI'm going to merge this. Since we'll have additional next steps for this, we can address any code deficiencies there.
Thanks all!
Comment #48
wim leers#41.2: I raised the same at #31. @gabesullice answered at #32.
Looks like Gabe's comment was inaccurate, since tests did pass?
Comment #49
gabesulliceThe comment wasn't inaccurate. The tests pass because that change was just an expansion of coverage—ensuring that the normalizer was properly registering itself with the serializer. We were doing it right before, there just wasn't coverage proving it before.
It also revealed a little "bug" in the test... notice that
JsonApiDocumentTopLevelNormalizer::classbecameNULLin the patch. This previous line was a "bug" because it was telling the normalizer to normalize the data into itself (!? lol :P ).While it might be technically out of scope, I think it's a little unfortunate that we lost that quick win.
Comment #50
wim leers#49: aha! I definitely read the patch too fast :) Could you open a new issue for that?
Comment #51
e0ipsoDo we need an issue for that? We're moving away from this pattern anyways. Besides the coverage expansion wasn't such. The
JsonApiDocumentTopLevelNormalizeris polymorphic and hence we decide the class to denormalize to at runtime. It's one of the abuses of the serialization that we had to make.Comment #53
pq commentedHi Guys,
Just found this thread while trying to work out why a custom Normalizer class I'd made was no longer being used after updating to the latest jsonapi dev release.
The class extended
FieldItemNormalizerand was declared in it's module's services.yml file with aname: normalizertag.Now the only way I can seem to get this to work again is to change the tag to
name: jsonapi_normalizer_do_not_use_removal_imminent, which obviously seems horribly wrong, but I can't seem to work out what the correct way of doing this is now that this change has been made.Any advice would be greatly appreciated.
Comment #54
e0ipso@PQ can you explain why you need a new normalizer? Maybe a computed field will work just as fine.
Comment #55
pq commentedHi @e0ipso, Thanks for getting back to me.
The normaliser is currently modifying the output of a custom field type. The field contains YAML encoded data (because reasons), and the normaliser decodes the YAML, sanitizes the data and outputs it as nested arrays (so it will appear in the output as nested JSON if that is the requested format).
If this isn't the preferred way to do this then I'm open to suggestions.
Comment #56
gabesullice@PQ, I think you should be able to do this at the TypedData data type level. We still respect/use normalizers at that very low level (IOW, the properties of your custom field).
Comment #57
gabesullice@PQ,
I realized my last comment was probably not very helpful. So I'll try again.
You probably have a
propertyDefinitions()method on a class extendingFieldItemBasethat returns something like$properties['your_yaml'] = DataDefinition::create('string')(if not, you can stop reading the rest of this and ping me in IRC/Slack lol cause I'd be interested to hear it and to put your unique case into this queue for others who might have the same problem).If my guess is right, then what you'll need to do is create a new custom data type for your YAML. So that in the end,, you can write
DataDefinition::create('custom_yaml')in that method instead.Then, what you'll do is modify your existing normalizer to apply to instances of that data type rather than at the level of the field item which contains your custom data.
Comment #58
gabesulliceFWIW, check out the implementation of the 'email' data type in core:
https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21TypedData...
All it does is extend from StringData, inheriting all its methods without overriding anything. The only difference is the annotation and a custom constraint.
Comment #59
pq commentedHi @gabesullice,
Many thanks for the info, I'll be having a go at your suggestion today if everything goes to plan.
My field class extends
Drupal\Core\Field\Plugin\Field\FieldType\StringLongItem, which does has thepropertyDefinitions()method. If I understand you correctly, I'll want to override to use a custom data type (that I also need to define).I'll post back here to let you know how it goes.
Comment #60
wim leersNote that this class is marked
@internal, meaning its implementation can change at any time.That's right! And then you'll be able to create a
@DataType-level normalizer. Few examples of those exist in core:\Drupal\serialization\Normalizer\PrimitiveDataNormalizer(for\Drupal\Core\TypedData\PrimitiveInterface, i.e. for several@DataTypeplugins). #2926508: Add DateTimeNormalizer+TimestampNormalizer, deprecate TimestampItemNormalizer: @DataType-level normalizers are reusable by JSON:API is adding one for@DataType=timestamp(\Drupal\Core\TypedData\Plugin\DataType\Timestamp).The added benefit is that this normalizer will work for both core's REST module and the JSON API module! Both will use the same (de)normalization logic :)
In fact, if you're willing to publish the code of your custom field type, I'm willing to provide the code for you, because I think it can serve as a valuable example in the change record.
Comment #61
pq commentedHi @Wim Leers,
Thanks for that, I'll probably be looking at making the suggested changes over the next few days as balancing tasks at the mo. I'll have a look at the possibility of publishing the code, although it will have to be so heavily redacted due to NDA's etc that the use case might no longer be apparent.
Comment #63
rpayanmMoving to Drupal core's issue queue.
I'm working on https://www.drupal.org/project/drupal/issues/3122113