Problem/Motivation

The links property is not always needed.
As such it is posing a proper overhead during generation and data transfers.

This started as JSON:API Extras issue, but due to complexity in how can that be overwritten from there, here we are :)...

Proposed resolution

Have a checkbox setting that will enable / disable links on a resource completely and so they are not generated at all.
In the specs it's stated what they should be with a proper format etc, but only if they are present.

In the specs it's not stated anywhere that they are a required property on all objects at all.

Where specified, a links member can be used to represent links. The value of each links member MUST be an object (a “links object”).

For example, an article’s comments relationship could specify a link that returns a collection of comment resource objects when retrieved through a GET request.

The optional links member within each resource object contains links related to the resource.

Remaining tasks

Patch, etc.
Benchmarks (payload size, generation time, used ram).

User interface changes

New checkbox to manage the functionality on a per resource level.
Maybe a single global one to manage the common default so we can spare developers clicking...

API changes

None.

Data model changes

New property expected on the configurable resource types.

Release notes snippet

TBD.

Comments

ndobromirov created an issue. See original summary.

ndobromirov’s picture

Issue summary: View changes
ndobromirov’s picture

Title: Configuration to enable / disable links property per entity resource type. » API to enable / disable links property per entity resource type.
Project: JSON:API Extras » JSON:API
Version: 8.x-3.x-dev » 8.x-2.x-dev

After a hacky PoC on this locally my total payload for a node (by ID) resource page dropped by about ~33%: 364 -> 246 kb (plain)
I have not done any benchmarks on the page generation times as of now. I will pass on the same hacky solution as a patch so anyone can at least test for himself.

What I've done is just commend-out everything that jsonapi module does for links generation in a non-error related context.

From what I saw, this might not be easy to do from jsonapi_extras perspective, as it does nothing in relation to the links generation and has little to no control for preventing that as well. For that reason only I am moving this up to jsonapi module's issue queue.

ndobromirov’s picture

Here are AB tests on the same page with and without the change.
No page cache.
No dynamic page cache.
No render cache.

Slowest request is 20% faster.
Mean request time is 10% faster.
Bytes transferred are 33% lower.

Patch is cumming shortly...

Running AB against apache on 81, as I have a varnish running on 80...
BEFORE:

DIRECTORY$ ab -n 10 -c 1 http://HOSTNAME:81/api/node/page/e8991086-41b6-4056-afd9-efbe7f0fb2dd
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking HOSTNAME (be patient).....done


Server Software:        Apache/2.4.18
Server Hostname:        HOSTNAME
Server Port:            81

Document Path:          /api/node/page/e8991086-41b6-4056-afd9-efbe7f0fb2dd
Document Length:        371618 bytes

Concurrency Level:      1
Time taken for tests:   22.155 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      3744650 bytes
HTML transferred:       3716180 bytes
Requests per second:    0.45 [#/sec] (mean)
Time per request:       2215.524 [ms] (mean)
Time per request:       2215.524 [ms] (mean, across all concurrent requests)
Transfer rate:          165.06 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:  2168 2215  81.1   2196    2444
Waiting:     2155 2203  80.8   2183    2431
Total:       2168 2215  81.1   2196    2444

Percentage of the requests served within a certain time (ms)
  50%   2196
  66%   2198
  75%   2205
  80%   2205
  90%   2444
  95%   2444
  98%   2444
  99%   2444
 100%   2444 (longest request)

AFTER:

DIRECTORY$ ab -n 10 -c 1 http://HOSTNAME:81/api/node/page/e8991086-41b6-4056-afd9-efbe7f0fb2dd
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking HOSTNAME (be patient).....done


Server Software:        Apache/2.4.18
Server Hostname:        HOSTNAME
Server Port:            81

Document Path:          /api/node/page/e8991086-41b6-4056-afd9-efbe7f0fb2dd
Document Length:        249302 bytes

Concurrency Level:      1
Time taken for tests:   20.262 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      2521490 bytes
HTML transferred:       2493020 bytes
Requests per second:    0.49 [#/sec] (mean)
Time per request:       2026.250 [ms] (mean)
Time per request:       2026.250 [ms] (mean, across all concurrent requests)
Transfer rate:          121.52 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:  1998 2026  20.4   2032    2056
Waiting:     1989 2018  20.2   2023    2046
Total:       1998 2026  20.5   2032    2056

Percentage of the requests served within a certain time (ms)
  50%   2032
  66%   2035
  75%   2041
  80%   2054
  90%   2056
  95%   2056
  98%   2056
  99%   2056
 100%   2056 (longest request)
ndobromirov’s picture

StatusFileSize
new5.75 KB

Here is the promised patch.

It differs a bit from my local tests, as latest dev and latest stable from 20th of February seems to be diverging a bit, but you can see the idea and the results should not be that different (unless I've missed some spot to comment out).

ndobromirov’s picture

Issue summary: View changes
ndobromirov’s picture

StatusFileSize
new4.27 KB

Same patch (logically), smaller diff size.

e0ipso’s picture

In the specs it's not stated anywhere that they are a required property on all objects at all.

That is true, however there is strong consensus on the JSON:API team to require this in the Drupal implementation.

This started as JSON:API Extras issue, but due to complexity in how can that be overwritten from there, here we are :)...

We should move this back to the JSON:API Extras issue queue and work on the implementation. I suspect the "new" links collector can be used for this. Just like jsonapi_hypermedia can add links we should be able to remove them. JSON:API Extras will be the place to add the configuration (a single on/off toggle for all resource types, like the count feature).

wim leers’s picture

my total payload for a node (by ID) resource page dropped by about ~33%: 364 -> 246 kb (plain)

How much difference is there with gzip or brotli enabled?

Just like jsonapi_hypermedia can add links we should be able to remove them

Exactly. For this use case, I think we can have a jsonapi_hypomedia module? 😆 (hyper vs hypo).

ndobromirov’s picture

I am not saying to remove them. This is just the hacky PoC that I can live with for now (urgent stuff)...

Just like jsonapi_hypermedia can add links we should be able to remove them.

But essentially this is just additional overhead. Why add something and then remove it :(. In the end it is going to be slower not having the links in...

It's ok to move it back to jsonapi_extras if it could be done from there.

@Wim, the 33% plain translate to ~16% gzipped. Dropped 34.3 -> 28.7 kb.

ndobromirov’s picture

Project: JSON:API » JSON:API Extras
ndobromirov’s picture

Version: 8.x-2.x-dev » 8.x-3.x-dev

As agreed moving back to extras module.

wim leers’s picture

#10: interesting, that's bigger than I thought. Thanks for letting us know!

ndobromirov’s picture

StatusFileSize
new5.96 KB

Just dumping this here as well, continuation of the previous patch. Ignore it.

OlgaRabodzei’s picture

Hi!

I attach a patch, which provides possibility to enable / disable "links" objects per resource.

However a top level "links" remains a part of the response.
I think enable / disable option for optional top level elements like "links" and "jsonapi" should be in another configuration, which would be more general. Imho such configuration should be on `admin/config/services/jsonapi/extras` page with path prefix settings.

OlgaRabodzei’s picture

Status: Active » Needs review
ndobromirov’s picture

There are 2 problems touched by this issue
1. PHP time spent generating URLs.
2. Response size in bytes, caused by excessive links in the response.

As it is, the patch from #15 solves only point 2, by adding some overhead to do the clean-up.

OlgaRabodzei’s picture

StatusFileSize
new6.77 KB

Here is an updated patch. I forgot to hide "links" checkbox on the configuration form, when a source is disabled.

OlgaRabodzei’s picture

And yes, Nikolay, I understand, that this patch doesn't make influence on the data generating performance.
But as it was mention in #8 comment, this is "a consensus on the JSON:API team to require this in the Drupal implementation".
So the only thing what we can do (without changing jsonapi module itself) is to play with the response view.

And I find the response less messy without all those links. Especially when it's read only endpoint.

e0ipso’s picture

I'm inclined towards allowing this. It seems a big enough difference to justify the downsides.

@ndobromirov is on point. I'd like to explore more before going for the second best solution (#18), which will provide a tangible win. Do you think we can decorate the link collector service to avoid generating the configured links? If not, do you think we can modify the collector to make this behavior possible? If we do that we can avoid generating the links in the first place.

e0ipso’s picture

I'm inclined towards allowing this. It seems a big enough difference to justify the downsides.

@ndobromirov is on point. I'd like to explore more before going for the second best solution (#18), which will provide a tangible win. Do you think we can decorate the link collector service to avoid generating the configured links? If not, do you think we can modify the collector to make this behavior possible? If we do that we can avoid generating the links in the first place.

ndobromirov’s picture

@e0ipso for better or worse it's not a single place where the links collection class is used, and the links are generated right before instantiating the piece.

To have this implementable I think we need to have some things added / extended in JSON:API:
1. Factory method / service for link collections to be used through the code. That will create either a links collection or a NULL links collection based on the configuration. It will allow for more ways of creating the link collections, so we could accommodate this use-case. JSON:API addition.
2. Callback based construction and not a list of links one, so we can postpone link computation to a later moment, where we could respect configuration or anything.
3. Refactor existing code.
4. Consider API breaks...

ndobromirov’s picture

I will have to revive this for core's 8.7.11+ soon...

ndobromirov’s picture

This can happen if we set isLocatable on all resource types to FALSE.
I am not saying we should, but if we expose this as resource type setting in JSON:API Extras it is 95%+ achievable.
Essentially preventing any links generation from JSON:API.
This will, in the most common case, prevent any links being generated and even if they are, it will be an empty set of links.
It will still be slower than the hack from #7 but a more cleaner solution overall.

Note that I am basing this on JSON:API from core 8.7.

If you know there is a better way in 8.8 I am open for suggestions, but at the moment I will work toward a patch with the setting on the resource types that will manage the isLocatable flag and from there integrate that into the configurable resource type altogether.

bbrala’s picture

Status: Needs review » Needs work

but at the moment I will work toward a patch with the setting on the resource types that will manage the isLocatable flag and from there integrate that into the configurable resource type altogether

Any progress on this? :)

nils.destoop’s picture

I re-rolled the patch in #7 for 9.2

bradjones1’s picture

I think actually this is better handled in core, if we have @bbrala's attention on this issue. See related.

ro-no-lo’s picture

Are there still any arguments left to have a switch in core to remove all links and their respective generation, if the use case do not need them? Even if the Drupal Json:Api Core team has a strong standing for providing these links in the output, if the individual website services don't need them, why generating and outputting them against their will and need? It seems like we "fight" over an optional JSON:API Refernece feature which will require only a few if conditions here and there.

I stumbled across this 4 years old issue, because I also want to remove them for our vue app. https://www.drupal.org/project/jsonapi_extras/issues/3311150

ptmkenny’s picture

@ro-no-lo My understanding is that the reason the links are included without any option to remove them is because that is what is required by the formal JSON:API specification, and the core module aims to provide 100% support for the specification. (See this comment in response to my question about removing links for NULL entity references.)

I understand the case for forcing the links always (if you say the module adds support for a specification, it should follow that specification), but at the same time, your use case and mine could really benefit from slimming down the responses by removing the links, and I am sure there are other sites in the same boat. So even if it violates the specification, I think this would be a very useful option.

e0ipso’s picture

I think this feature is often an anti-pattern. I am not very supportive of promoting this practice. Lack of usage of hypermedia capabilities is one of the biggest factors in maintenance problems in decoupled applications.

If your server is using a compression algorithm the difference in the response length should be negligible. If not, that will be a way bigger win across the board than removing some links in JSON:API responses.


That being said, I will defer final decision to @bbrala.

e0ipso’s picture

Also note that the initial report was created before we improved internal caching of normalizations, so the generation of links should not be as big of a problem nowadays. Can anyone corroborate this?

wim leers’s picture

I agree with @e0ipso.

If you want to modify all JSON:API responses to not contain any links, that's easy enough to achieve:

  1. create a new jsonapi_linkless contrib module
  2. add a KernelEvents::RESPONSE event subscriber that runs after jsonapi.resource_response.subscriber (which has priority 128), with a priority lower than Dynamic Page Cache's (100), to ensure the results are cached by Dynamic Page Cache — so any priority in the [101, 127] range.
  3. omit all links from the response in that event subscriber, if the response is a JSON:API response
ro-no-lo’s picture

@ptmkenny: The jsonapi.org states that the resource links are optional. https://jsonapi.org/format/#document-resource-object-links

@e0ipso: currently we develop a vue3 app and these modern app come with a lot of packt vuex stores. With more and more entities loaded into the browser system everything grows. I do not have enough expertise to know what will happen, when every call to the json:api and lists of entities come with a 30% overload of unused data. We may be not experienced enough to build such a hypermedia usage into the system, but thats were we are currently. My concern is, that the memory usage is growing for processing the JSON responses from the server, even though it could be smaller. Time will tell.

@wim-leers: Thanks for the tipp. I will consider it, when we run into memory problems.

e0ipso’s picture

My concern is, that the memory usage is growing for processing the JSON responses from the server

I think this may not be a problem as per #31.

currently we develop a vue3 app and these modern app come with a lot of packt vuex stores. With more and more entities loaded into the browser system everything grows. I do not have enough expertise to know what will happen, when every call to the json:api and lists of entities come with a 30% overload of unused data. We may be not experienced enough to build such a hypermedia usage into the system, but thats were we are currently.

I understand that, and I empathise with the situation. However, as a maintainer this is not something I want to prioritize in this module. But, again, I will follow what @bbrala decides since he is carrying the baton of this module.

ro-no-lo’s picture

For everybody interested. I managed to put a module together with the hints Wim Leers gave me. https://www.drupal.org/project/jsonapi_links If you find stuff you could optimize, do so and create a merge request.

allahnoor turab’s picture

26 and 18 are not applying to drupal version 9.5 for me

ab.shakir’s picture

Re-roll of #14 for D9.5.8.

e0ipso’s picture

Status: Needs work » Fixed

Thanks for all the contributions! It sounds like ro-no-lo provided a successful solution to this in #35.

Closing.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.