Comments

gabesullice created an issue. See original summary.

gabesullice’s picture

Status: Active » Needs review
StatusFileSize
new34.2 KB

Status: Needs review » Needs work

The last submitted patch, 2: 2987608-2.patch, failed testing. View results

wim leers’s picture

I'm less convinced by this change actually. Can you explain the rationale?

gabesullice’s picture

The rationale is simply that it's a necessary step to get rid of the request handler. I should have already put this in the [META] issue, this issue comes with the understanding that deserialization will later be moved to a route enhancer.

Why?

Today, we have a small bit of overhead on every request.. the first line of RequestHandler::handle is $parsed_entity = $this->deserialize($request, $resource_type);. That runs even for GET requests (at least as long as it's not already cached). Moving to specific methods means we can avoid that small step.

After that's done and arguments are being dynamically resolved, we'll be able to move this back out of EntityResource into a route enhancer so it'll run for only methods with a $parsed_entity argument. We can't do that now without adding complexity to the RequestHandler.

wim leers’s picture

   if ($request->isMethodSafe(FALSE)) {
     return NULL;
   }

means that ::deserialize() returns early for GET requests.

we'll be able to move this back out of EntityResource into a route enhancer

Ah, that would make it worthwhile indeed!

IOW: this is a step on the path towards moving the mapping of a request body (through deserialization) to a controller parameter. Correct?

gabesullice’s picture

[code] means that ::deserialize() returns early for GET requests.

Yep, and this is somewhat of a workaround for that fact that we don't currently have a way to call deserializejust for the routes that need it.

IOW: this is a step on the path towards moving the mapping of a request body (through deserialization) to a controller parameter. Correct?

Exactly.

gabesullice’s picture

Status: Needs work » Needs review
StatusFileSize
new38.31 KB

I was expecting #2987606: Remove config mutation tests from EntityResourceTest to be committed fairly easily, so I let this issue depend on it. That's not entirely necessary though.

So instead of postponing this issue on that one, I rerolled this patch to fix the config related tests rather than just assume that they're gone.

Status: Needs review » Needs work

The last submitted patch, 8: 2987608-8.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

gabesullice’s picture

Status: Needs work » Needs review
StatusFileSize
new39.58 KB
new1.19 KB
new3.64 KB

This should pass 🤞

There are two interdiffs, one shows the change to make the tests pass, the other is code standard fixes. Both are part of the patch.

wim leers’s picture

Status: Needs review » Needs work
  1. +++ b/jsonapi.services.yml
    @@ -149,11 +149,9 @@ services:
    -    public: false
    
    +++ b/tests/src/Kernel/Controller/EntityResourceTest.php
    @@ -163,14 +164,7 @@ class EntityResourceTest extends JsonapiKernelTestBase {
    +    $this->entityResource = $this->container->get('jsonapi.entity_resource');
    

    Do we really need this to be public?

    I think this test is the only reason we make it public? If so, let's just define a test-only service in the test's setUp() that is a public alias of this private service. That will achieve the same!

  2. +++ b/tests/src/Kernel/Controller/EntityResourceTest.php
    @@ -551,8 +538,9 @@ class EntityResourceTest extends JsonapiKernelTestBase {
               'type' => 'article',
    -          'title' => 'PATCHED',
    -          'field_relationships' => [['target_id' => 1]],
    +          'attributes' => [
    +            'title' => 'PATCHED',
    +          ],
    

    Nice bugfix!

  3. +++ b/tests/src/Kernel/Controller/EntityResourceTest.php
    @@ -666,52 +663,35 @@ class EntityResourceTest extends JsonapiKernelTestBase {
    -  public function testDeleteIndividualConfig() {
    

    Shouldn't this be moved to jsonapi_extras?

  4. +++ b/tests/src/Kernel/Controller/EntityResourceTest.php
    @@ -725,14 +705,18 @@ class EntityResourceTest extends JsonapiKernelTestBase {
    -   * @dataProvider patchRelationshipProvider
        */
    -  public function testPatchRelationship($relationships) {
    +  public function testPatchRelationship() {
    
    @@ -749,70 +739,46 @@ class EntityResourceTest extends JsonapiKernelTestBase {
    -  public function patchRelationshipProvider() {
    -    return [
    -      // Replace relationships.
    -      [[['target_id' => 2], ['target_id' => 1]]],
    -      // Remove relationships.
    -      [[]],
    -    ];
    -  }
    

    Why this change? (It seems to be removing valuable test coverage?)

  5. +++ b/tests/src/Kernel/Controller/EntityResourceTest.php
    @@ -749,70 +739,46 @@ class EntityResourceTest extends JsonapiKernelTestBase {
    -   * @dataProvider deleteRelationshipProvider
        */
    -  public function testDeleteRelationship($deleted_rels, $kept_rels) {
    +  public function testDeleteRelationship() {
    ...
    -  public function deleteRelationshipProvider() {
    -    return [
    -      // Remove one relationship.
    -      [[['target_id' => 1]], [['target_id' => 2]]],
    -      // Remove all relationships.
    -      [[['target_id' => 2], ['target_id' => 1]], []],
    -      // Remove no relationship.
    -      [[], [['target_id' => 1], ['target_id' => 2]]],
    -    ];
    -  }
    

    Why this change? (It seems to be removing valuable test coverage?)

gabesullice’s picture

Status: Needs work » Needs review

1. Instead of making an alias, I just instantiated it in the setUp method.
2. :)
3. I added the test back and marked it skipped with a note to move it in #2977659: Spec Compliance: POST|PATCH|DELETE on relationships should respect arity rules.
4+5. Added this coverage back in. #2977659: Spec Compliance: POST|PATCH|DELETE on relationships should respect arity rules will also add much more comprehensive coverage for all these scenarios (including cache info, status codes, etc).

gabesullice’s picture

StatusFileSize
new42.51 KB
new8.75 KB

Files actually attached this time.

wim leers’s picture

  1. +++ b/src/Controller/EntityResource.php
    @@ -963,4 +962,69 @@ class EntityResource {
    +  protected function deserialize(Request $request, ResourceType $resource_type) {
    
    +++ b/src/Controller/RequestHandler.php
    @@ -72,72 +57,10 @@ class RequestHandler {
    -  protected function deserialize(Request $request, ResourceType $resource_type) {
    

    👍 Confirmed these are identical before & after!

  2. +++ b/tests/src/Kernel/Controller/EntityResourceTest.php
    @@ -719,94 +737,156 @@ class EntityResourceTest extends JsonapiKernelTestBase {
    -  public function testPatchRelationship($relationships) {
    +  public function testPatchRelationship() {
    ...
    -  public function patchRelationshipProvider() {
    ...
    +  public function testDeleteRelationship() {
    ...
    -  public function deleteRelationshipProvider() {
    

    You restored this test coverage, but it looks *very* different now. Is that truly necessary? It makes this patch much harder to review, and the interdiff twice as large.


I've just applied this patch plus #2987609-21: Rename the entity parameter from the entity type ID to 'entity' for all routes plus #2987610-10: Remove RequestHandler class and service and add EntityResource methods to each route definition, to see the end result.

  1. Select tests seem to be passing 👍
  2. 20 files changed, 431 insertions(+), 448 deletions(-) is not as exciting a diffstat as I'd hoped … 😞
  3. And I've yet to see this, and am having some doubts:
    we'll be able to move this back out of EntityResource into a route enhancer

    Ah, that would make it worthwhile indeed!

    How can a route enhancer deserialize a request body? AFAIK this is not possible?

gabesullice’s picture

Just responding to the last bit ATM...

How can a route enhancer deserialize a request body? AFAIK this is not possible?

See JsonApiParamEnhancer. We already access the request's query parameters and denormalize them into a Filter object.

We would pretty much do the same as we do there. I.e., access the request, get its content, then add the parsed value as a default route argument. That default would be treated as a parameter to EntityResource::{method}() as resolved by the arguments resolver.

After thinking about it more though, I think we may be able to do that today, not tomorrow. Previously, I didn't see a way to take the route enhancer approach while the RequestHandler still existed... and I didn't know how to get rid of RequestHandler while it still did deserialization. Hence I wanted to do it in 3 steps. Move deserialization, remove the request handler, then put deserialization in a route enhancer. BUT, I have an idea that I'd like to try out that will get rid of this step. Stand by!

gabesullice’s picture

StatusFileSize
new53.5 KB
new23.13 KB

Okay, here's the approach as discussed in #15. That cut the patch size in half! And we managed to make more deletions than insertions! Breaking it down by src/test directories, it looks even better:

$ git df 8.x-2.x --shortstat src/
 3 files changed, 85 insertions(+), 110 deletions(-)
$ git df 8.x-2.x --shortstat tests/
 7 files changed, 39 insertions(+), 23 deletions(-)

So, what happened?

I used a route enhancer to deserialize the entity into a default parameter and then accessed that deserialized value in the RequestHandler before passing it to EntityResource.

That's just what we need to do in order to set up #2987610: Remove RequestHandler class and service and add EntityResource methods to each route definition.

With this approach we barely need to touch EntityResource at all. The only thing I did was make it a bit more consistent because using route info already available to better understand when the body actually needs to be deserialized and when it doesn't.

Unfortunately, I had to make jsonapi.serializer_do_not_use_removal_imminent public... I spent way too long trying to figure that one out and finally just gave in. I can't figure out how to alias it in a BrowserTest because $this->container->setAlias isn't a available. I couldn't simply instantiate it myself because we need the serializer to be injected with all the normalizers and I can't get those tagged services.

This public/private thing seems to be a recurring issue. I think we need to take some time to figure out a good solution for this as a general solution rather than struggling with this over and over.

Finally, the only failing tests I know of ATM, are the unit tests for JsonApiParamEnhancer, again, because of the private service issue. I honestly think we can remove that test class though, but I wanted to get others' opinion first.

Status: Needs review » Needs work

The last submitted patch, 16: 2987608-16.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

gabesullice’s picture

StatusFileSize
new1.69 KB
new23.13 KB

CS fixes.

gabesullice’s picture

Status: Needs work » Needs review
StatusFileSize
new4.37 KB
new27.5 KB

And here it is with tests/src/Unit/Routing/JsonApiParamEnhancerTest.php removed if no one objects to removing it.

wim leers’s picture

Status: Needs review » Needs work
Issue tags: +needs profiling

#15: Ahhhh, of course! Makes sense.
#16: Woah! 😮 Now that is an interdiff :D It does look like some other patch got entangled in there though.
#19: I first wanted to bring back the test coverage. But then I realized that the existing test coverage was literally just verifying that a specific normalizer would be called, and the logic in HEAD for JsonApiParamEnhancer was in fact already hardcoding a specific normalizer for a specific purpose. So the test coverage was not really testing anything. I agree that this test coverage has served its purpose (dating back to JSON API's early days, added in May 2016, commit 2a2b593!)


  1. This needs to be rebased now that #2977659: Spec Compliance: POST|PATCH|DELETE on relationships should respect arity rules is in.
  2. +++ b/jsonapi.services.yml
    @@ -104,12 +103,12 @@ services:
    -    arguments: ['@serializer.normalizer.filter.jsonapi', '@serializer.normalizer.sort.jsonapi', '@serializer.normalizer.offset_page.jsonapi']
    +    arguments: ['@jsonapi.serializer_do_not_use_removal_imminent']
    

    I wonder if this has a performance impact.

    In HEAD, we're instantiating 3 normalizer services during routing.

    With this, we're instantiating the entire JSON API serializer and hence all JSON API normalizers … but also core's serializer and all of its normalizers.

    From 3 services in the critical routing path to many dozens.

    That means this approach definitely needs profiling.

  3. +++ b/jsonapi.services.yml
    @@ -104,12 +103,12 @@ services:
    -    arguments: ['@router.no_access_checks', '@url_generator']
    +    arguments: [null, '@url_generator']
    

    🙃😵

gabesullice’s picture

Status: Needs work » Needs review
StatusFileSize
new27.21 KB

1. Rerolled.
2. Hm. I'll need some guidance on how to do that.
3. I don't understand those emojis :P did you want something to change, or are you just remarking on its insanity?

gabesullice’s picture

gabesullice’s picture

StatusFileSize
new2.03 KB
new26.85 KB
gabesullice’s picture

Title: Move deserialization from RequestHandler to EntityResource » Move deserialization from RequestHandler to ParamEnhancer

Made the title more accurate.

gabesullice’s picture

Title: Move deserialization from RequestHandler to ParamEnhancer » Move deserialization from RequestHandler to JsonApiParamEnhancer

Again.

gabesullice’s picture

StatusFileSize
new26.86 KB

Rerolled.

gabesullice’s picture

StatusFileSize
new4.4 KB
new27.41 KB

#20.2 pointed out that the JsonApiParamEnhancer is instantiating normalizers for every routing invocation. This patch might exacerbate that because it instead instantiates the entire serializer and all normalizers.

Rather than let this get blocked on that case and profiling, let's just lazy load the serializer under all circumstances.

wim leers’s picture

Status: Needs review » Needs work

Rather than let this get blocked on that case and profiling, let's just lazy load the serializer under all circumstances.

Pragmatism FTW! 👍


  1. +++ b/jsonapi.services.yml
    @@ -4,7 +4,6 @@ parameters:
    -    public: false
    

    I'd like to see this change removed from the patch. Is that still not possible?

  2. +++ b/jsonapi.services.yml
    @@ -104,7 +103,8 @@ services:
    -    arguments: ['@serializer.normalizer.filter.jsonapi', '@serializer.normalizer.sort.jsonapi', '@serializer.normalizer.offset_page.jsonapi']
    

    What is again the problem with keeping these as-is?

    Or, put differently, why are these even normalizer services at all?

  3. +++ b/src/Routing/JsonApiParamEnhancer.php
    @@ -3,47 +3,45 @@
    +      $this->serializer = $this->container->get('jsonapi.serializer_do_not_use_removal_imminent');
    

    Ah, this means it's not possible.

gabesullice’s picture

Status: Needs work » Needs review

2. At first, it was because we also needed the serializer for deserializing the incoming entity or resource identifiers. Therefore it seemed excessive to individually inject every one of those. Later, it's because it turns out that in many instances, we don't need them at all. Since this is on the critical path, perhaps it's better to avoid loading anything. OTOH, maybe that a micro-optimization.

They're normalizers because we're taking a particular wire encoding (the query parameter string), decoding it into an array, and producing a non-scalar object. Which is exactly what the serialization system is designed to do 🙂 (decode -> denormalize or normalize -> encode).


This was set to "Needs work", but I don't see any word to be done. Am I missing something?

The last submitted patch, 18: 2987608-18.patch, failed testing. View results

wim leers’s picture

It was NW for #29.1, and kinda for #29.2.

#30.2: yes, you could theoretically argue that way that this is denormalization for the reasons you cite. But … only barely so. According to the same line of reasoning, \Drupal\Core\ParamConverter\ParamConverterInterface::convert() should be removed in favor of a denormalizer too. Because really, this is mapping information in a URL to a value object. The opposite direction (normalization) never occurs. I think it's pretty clear this is not using the Symfony serialization system fore the intended purpose.
Another reason you cite is to not load anything unless they're actually needed, but … that's only because we're using the serialization system at all. Usually, I'd argue that this means that new things can be added in the future (in this case: new parameters). But that's not even true, since \Drupal\jsonapi\Routing\JsonApiParamEnhancer::enhance requires a mapping from URL query arg name to denormalization class to be hardcoded anyway!

If we'd remove these normalizer services, we'd have less weirdness, and compared to #28, we'd be able to not inject normalizer services nor pass the entire container. Passing the entire container is always frowned upon.

Additionally, all this logic only makes sense (only runs) for JSON API collection resources anyway. Just grep for _route_params and _json_api_params. So rather than implementing \Drupal\Core\Routing\EnhancerInterface, which is executed at runtime (because the signature is public function enhance(array $defaults, Request $request);), it'd be much more logical to run this in \Drupal\jsonapi\Controller\EntityResource::getCollection() instead. That's the only place where it's being used. And hence it's the only place that should be slowed down by this.

But doing all that would be a much bigger refactoring. It would remove the bulk of JsonApiParamEnhancer::enhance(). It'd remove what that was originally designed for (i.e. "the JSON API params", meaning filter, sort and page), but the deserialization of the request body would still be left to be done. So the serialzier would still need to be injected.


So, rather than insisting things be done even better, I think this patch is already better than what's in HEAD. There's a net negative diffstat. And it directly paves the path for #2987610: Remove RequestHandler class and service and add EntityResource methods to each route definition, which will result in the RequestHandler class being deleted. Core REST will be handling its request body deserialization in its RequestHandler class, JSON API will be handling it in a param enhancer. Neither is perfect.

My chief concern is that this may get in the way of #2958554: Allow creation of file entities from binary data via JSON API requests, which is IMHO far more important than this because it brings an important new capability to JSON API. In core REST, this required changes to RequestHandler. But #2958554 is doing stuff with an alternative RequestHandler, just like core REST. So I think that before this can be committed, we need to land #2958554: Allow creation of file entities from binary data via JSON API requests (ideal case) or we need to be very certain that removing JSON API's RequestHandler won't get in the way. Hence keeping at Needs review.

gabesullice’s picture

My chief concern is that this may get in the way of #2958554. But #2958554 is doing stuff with an alternative RequestHandler, just like core REST. So I think that before this can be committed, we need to land #2958554... or we need to be very certain that removing JSON API's RequestHandler won't get in the way.

I'm very certain that it won't.

#2987610: Remove RequestHandler class and service and add EntityResource methods to each route definition removes the request handler entirely. It is replaced by route-specific calls to a service and specific methods.

#2958554: Allow creation of file entities from binary data via JSON API requests lives in a submodule and has its own RequestHandler which is just a service+method. It does not extend the JSON API request handler. Having been a guiding hand in the file upload issue, I can say that a clean separation of concerns was at the forefront of my mind precisely for this reason. I knew I wanted to remove the request handler.

And I know that I want to remove the request handler for cases like this one. Being able to "weave" in alternate JSON API routes because we don't have a hardcoded bottleneck in the RequestHandler was/is one design goal of #2953346: Define related/relationship routes per field, not dynamically (with route parameters that need validating) and #2987610: Remove RequestHandler class and service and add EntityResource methods to each route definition.

In fact, I have a chat record with @malik.kotob where we discussing this in relation to file uploads:

Gabe Sullice [9:55 AM]
That would be a good start :slightly_smiling_face: In the end, I think we'd want to inspect the resource type and make routes for each file field (but don't worry about that for now)
I _just_ did a similar thing here: https://www.drupal.org/project/jsonapi/issues/2953346

Given that the file uploads patch operates with a completely distinct handler and that it uses a subrequest to return its responses. I think we can safely proceed.

wim leers’s picture

Issue tags: -needs profiling

Per #28.

wim leers’s picture

Status: Needs review » Reviewed & tested by the community

I wrote in #32:

My chief concern is that this may get in the way of #2958554: Allow creation of file entities from binary data via JSON API requests, which is IMHO far more important than this because it brings an important new capability to JSON API. In core REST, this required changes to RequestHandler. But #2958554 is doing stuff with an alternative RequestHandler, just like core REST. So I think that before this can be committed, we need to land #2958554: Allow creation of file entities from binary data via JSON API requests (ideal case) or we need to be very certain that removing JSON API's RequestHandler won't get in the way. Hence keeping at Needs review.

@gabesullice wrote in #33:

I'm very certain that it won't.

But because it's not trivial to see, I wanted to answer this by proving it with passing tests: #2958554-80: Allow creation of file entities from binary data via JSON API requests is #2958554 + this patch + #2987610: Remove RequestHandler class and service and add EntityResource methods to each route definition. It passes tests. Therefore there cannot be any doubt in anybody's mind that this is safe to do. Hence: RTBC, and committing :)

  • Wim Leers committed 03dd1d4 on 8.x-2.x authored by gabesullice
    Issue #2987608 by gabesullice, Wim Leers: Move deserialization from...
wim leers’s picture

gabesullice’s picture

🎉

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.