[FEATURE] Respect the provided maximum page[limit] value

This was a conscious decision. There should be an upper limit to prevent DDOS-ing your own site inadvertently.

If you need more than 50 records, you should really think about making parallel paginated requests of 50 items.

I'm marking this is as feature request to be considered in the future. This would depend on much more robust configuration management for the module as a whole (which is in the roadmap), so this is probably a low priority item for now. I'm not closing because there are probably valid/safe use cases for increasing the maximum page size, albeit rare ones.

Is this already implemented? I don't want to execute 3 calls to fetch a list of items (which consists of 130 item). It would be nice to have a feature "Give me list of all entitties"?

@robin.ingelbrecht how would you prevent the performance impact of loading so many entities at once in a cold cache request?

Allowing this in the module could have some serious downsides to it. Any ideas on how to come about this?

I'm not requesting the whole entity, I'm only requesting 2 fields (via the field param), so this would not be a huge performace issue...

Personally I think it should be up to the developer wether or not to restrict the result set? Now the json api module forces you to do multiple requests above 50 results. What do you think?

It still is up to the developer. You can swap the service with your own class and have the number be 2000000000 instead of 50. What we are considering here is supporting that in this module.

The performance penalty (on the server) for loading 20000 or 150 entities will still be there even if you only select a single field. Unless there is an idea about how to overcome that, I don't like the idea of unrestricted collections.

You're right about the performance penalty on the server and I follow your concern about unrestricted collections, but isn't 50 a little bit low then? Chances are that 90% of the people that use this api would like to retreive a list bigger than 50 items. In my opinion it is quite a hassle to define your own service and class just to raise that number?

For now, I'll just swap the service and raise it to 150 ;).
Thanks for your awesome work though!! I attended your session on DrupalCon Dublin and started using this module right away, it's a revelation :)!

50 can be low and high, depending on the site or even on the bundle. There is no way of telling. It's much simpler to make multiple paginated requests.

@robin.ingelbrecht thank you for the kind words. Also, excuse me for the brevity above. I was on the phone.

The biggest point here is that we need to set a max limit somewhere. I see your point that 50 is too small for you. The problem is that the drupal module needs to have any possible drupal site in mind, and therefore has to play safe. Playing safe is not always ideal for other site installations.

With that in mind, I think that this should be configurable per resource. Right now we don't have anything in place to provide configuration per resource, but @janstoeckler may be working on that for #2782755: [FEATURE] Allow changing names and disabling for fields. Once that is in, we can reuse those structures to have site owners override the 50 items limit per resource.

I too was a bit surprised that no matter how high I set the page[limit] value, it was always capped at 50.

The provided patch will make sure that whatever value is provided in page[limit], it will be used. Even if it's above 50. If the limit is not provided, the value is still set at 50.

Updating the test to comply with the changes to how the limit parameter works with this patch.

This may end up being a risk and a potential vulnerability. I don't think it's a good idea to commit any code that allows a 3rd party user bring the server down by requesting 100000 entities. If you think it's a good idea to have that in your site, you can swap the service with your own class, as explained in #7. Please, let's shift the discussion to how to do actually that.

If any of you end up implementing this people will appreciate any documentation on how to do it.

Thanks all!

This is silly. Set the default to 50, and you've covered your ass, but for gods sake let this be modified. There's no need to force someone to produce their own class just to change a variable because you think you've chosen a magically perfect number.

@endorn what is your use case that cannot be handled by swapping the service or by automated pagination by the consumer?

@e0ipso

So in my scenario, we have a react app that requests data from about 3000 entities at a time. Loading that takes about 3-4 seconds. It's a small user base, and these requests are made maybe a few hundred times a day.

If I have to split 3000 entities into 60 requests that's going to increase load time significantly for no benefit.

The bigger point is if you arbitrarily limit people without a good way, they just going to hack the module. Most people arn't going to try and figure out how to replace the service, they're going to edit OffsetPage.php and change public static $maxSize = 50; to public static $maxSize = 999;

@dagmar isn't the entire point of this module to give front end developers access to the data without touching the backend? If a dev understands something as complicated as overriding a service then he knows enough to just create routes that do a query and spit out json exactly how he needs it.

The other thing that really bothers me about this is that this module is supposed to be a drupal implementation of the JSON API spec. Those specs very clearly define page limits and offsets as configurable values. http://jsonapi.org/format/#fetching-pagination

I don't understand the logic behind breaking specification to limit functionality arbitrarily to protect the developers from themselves. There's a million (probably literally) other ways and places in drupal we can do something stupid and crash the server or let a third party crash our site. It's part of the responsibility of the developer to code so that doesn't happen.

@endorn by stating

If a dev understands something as complicated as overriding a service then he knows enough to just create routes that do a query and spit out json exactly how he needs it.

You are either oversimplifying what this module does or not understanding the data model traversal and filtering capabilities.

The other thing that really bothers me about this is that this module

Sorry to hear that free software bothers you.

Those specs very clearly define page limits and offsets as configurable values.

I cannot see that there. However they are configurable within the limits, we have tests to ensure that. I believe this id offtopic, since your disagreement is on the upper hard limit.

There's a million (probably literally) other ways and places in drupal we can do something stupid and crash the server

That's not a great argument. There are other things that are brittle, let's make this brittle too. Even if there are other places where that would happen, I don't have to maintain those (and the related hurdles).

In general I would say that you have a slightly aggressive way to discuss. The facts:

1. There is a rationale behind a decision you do not agree.
2. The maintainer has given you the reasons why this will not be implemented.
3. The maintainer has kindly given you the explanation on how to solve your problem in two different ways.
4. The maintainer has kindly asked to share documentation if you end down that route.

@e0ipso Sorry, I wasn't meaning to come off aggressive. This is just a passionate thing for me because I want Drupal to move in the direction of letting react and angular applications work with drupal smoothly via API. JSON API is a great spec for this, with many tools available, but with a hard coded limit of 50 results I won't be able to recommend this module to my react.js developer friends.

Your solution of swapping the service with your own class is difficult if you're a low to mid level D8 developer, but imagine being a react developer who's just trying to interact with D8 programmatically through a nice standard spec and being told you have to learn Symphony and expert level D8 module development just to return more than 50 results at a time.

In fact, I was in a training monday of drupal con where we had a class given by 4 kitchens of about 30 people working all day on a react & drupal application using this module. It was a simple todo application where you could vote each todo item up or down. After working all day the class was unable to get the app working because 30 people each liking 2-3 todo items each resulted in more than 50 results and 4 kitchens hadn't tested the app with a larger userbase.

All that being said, It's your module and you're right... if the maintainer says no, then I'm out of luck. I do want to thank you for contributing to the community even if the module won't work for me, and apologize again if I came off aggressive, it's just my passion for using drupal as a back end for react web apps coming through.

@endorn no worries.

You seem to be skipping the recommendation (which is what everyone I know is doing and the official recommendation) to do auto-paginated requests. Are there any issues with that?

FWIW I know for a fact that Luke (your trainer last Monday) is using auto-paginated requests in other projects.

@e0ipso Paginated requests would work great for something like a google search result, or a list of upcoming events etc...

The majority of my app projects deal with data analysis where we have 3-5,000 people at a time with demographic information and a person needs to quickly analyze them. For example, I might be planning a trip to Dallas Texas to meet with someone important, so while I'm out there I might need analyze the people we have in that region.. so I might load the Texas region with about 5,000 potential candidates and then see how many are within 50 miles of Dallas, which might filter me down to 1,000 people, then I can search for people who've been active with us in the past year which may take us down to 500, then I can look for people who have spent over $20,000 with us lifetime which would probably get me down to the 10 or so important people I can reach out to and schedule meetings with for my trip.

This is a simplified version of what I'm talking about, the app we built here has over 30 data points that can be filtered down like that. React is brilliant for this, one 3-4 second call to drupal and the user can instantly filter with no callbacks as much as he wants trying out whatever combination of filters he wants until he finds the exact data set he's looking for, all without hitting the webserver again. Sometimes in a particular large or important data set they can spend all afternoon working with this data, all without impacting the web server.

If I were to have to paginate the calls, either the initial call would have to be split up into 100 calls which would drastically add to impact and load time, or every time they wanted to try out a different filter I'd have to make a call back to the webserver, and even then the results would still need to be split up into multiple calls.

Here's another simpler example:

https://give.wvu.edu/

I built this react.js based navigation with a drupal back end. WVU Has over 300 different schools, groups, colleges, organizations, etc.. you can give to. Clearly you can't list 300 menu items, and paginating navigational links clearly isn't a good solution.

So we download all the menu structure as a json object and use it in a react app. We *could* paginate based on the category, but there's no real need to, pulling down 300 giving opportunities takes about 200 milliseconds and allows the user to search for what they're looking for instantly, instead of typing in a search term, hitting submit, waiting for the page load, looking at results realizing they messed up, searching for something else, etc... Just pulling down all 300 at once gives the user a better experience and lessens the impact on the web server.

I think it all boils down to:

100 calls which would drastically add to impact and load time

which I don't think is correct for a browser based application. And even if you do, yo can use something like https://www.drupal.org/project/subrequests for it.

For the skeptical, here is an exaggerated test (trying to push the point). You cannot see the dramatic degradation.

So for that to work, I'd have to know how many total results there are so I know how many requests to make.

So my react app would start with a request that does the query for all 5000 results, but just returns the count.
Realizes I'll need 100 seperate calls to get the data.
Makes the calls and drupal repeats the query from step 1 100 times (with limits of course)
combines all the results into a useable data set.

- or -

React app starts with a request that does the query for all 5000 results, and returns the data.

Which seems like the cleaner solution?

I wasn't trying to re-open the argument here, I was just pointing out a few use cases where a 50 limit did not make sense.

I get that there are work arounds, but If I'm going to go to the bother of creating a route to tell me what my total number of results are so that I know how many pages I'm going to have to separate into requests, then make those requests, or installing another module and using sub requests, I might as well just do an entity query and return Json results from a route and be done with it.

Really not trying to sound aggressive again, but please keep in mind we're talking about doing all this stuff, and installing other contrib modules, just because there's a hard coded cap of 50 in the module and it just seems really silly to me. Not trying to fight about it, and I'm pretty sure I wont change your mind. I'm sure you built this module with a specific use case in mind and I'd bet it fits that perfectly. I just think with the 50 cap, application developers are going to have to use something else.

@endorn

I really understand your concern and your desire for a change. Back in #3 I postponed this issue because I could understand your perspective, but it also didn't seem like an appropriate issue to take on at the time. However, for all that this module provides, we have managed to keep it to ZERO configuration with sane defaults. It seems like a mistake to break that rule for something that is so subjective.

From your perspective 50 items is far too small for 1 request and is not sane™. However, from ours, any more would not be sane.

Let me try to illustrate why:

In one of your first comments, you said you would need to override it to 999, so let's just take that as an example.

If it's 999 requests for nodes with an include for author entities (let's say so you can display the author's first name), that means the module will need to load 999 * 2 entities into memory. That's approximately 2000 objects in memory. If it's a hefty node type, this could easily cause PHP to run out of memory and crash. This is not just speculation, but a reality.

Moreover, we would be required to serialize each of those entities as well. I.e. convert them to into the right JSON API structures (meaning it's harder than just a pure json_encode of the entities. This takes non-trivial time.

As a frontend developer, especially in React, I'm sure you can appreciate the necessity of page "responsiveness." That is, it should feel fluid and intuitive and it should not take excessive amounts of time to begin rendering useful layouts.

Drupal is not always a high performance application. Serializing even a smaller number of entities could push requests up beyond 10 seconds to fulfill. I'm sure you'll agree, this would not be an ideal user experience as you would have no data to render. Even if the limit were higher, you would end up implementing pagination yourself.

Moving the limit UP, would not be sane. It would cause behavior that reflected poorly on the module and poorly on Drupal itself.

Truly, this is a question of application developer competency (I'm sorry to put it so bluntly).

Even if one doesn't want to implement "next" and "previous" buttons on the frontend (instead just rendering results further and further down the page), the proper approach to this problem would be to _paginate_ one's requests. As the application gets the first set of items, it should render them. Then when it gets the next set of items, it should render them below the ones it already has retrieved. This is the whole point of a reactive application. As the data is updated or fetched, the application re-renders the page to accommodate it.

I recognize that implementing pagination is not easy. But that is why we have NPM. If the application developer is implementing pagination over and over, they're doing it wrong™. That developer should be using a JS library on the front-end such that the logic of the HTTP requests is completely opaque to the rest of the application. That library should use generators/promises to resolve the entirety of the request regardless of the pagination size and it should do so without leaking the HTTP logic into the greater application.

This is where we come in as the module maintainers, the spec is clear about pagination: http://jsonapi.org/format/#fetching-pagination. A server MAY choose to limit requests. BUT, the MUST provide all the required pagination links. This is what we do.

The spec is written this way not as a guide to developers to achieve their very specific tasks, but so that generic clients can be written that are completely blind to the implementation. Just as on the server side, we have implement a JSON API that is completely oblivious to the entity types and structures of any particular site.

Whether the limit is 5, 50, or 5000, the client library should be agnostic about it as we are to the actual data that we expose.

Put simply, the debate is not about the "right" limit or even if a limit should exist at. But simply about best-practices. If one is not writing a client in such a way that paginated results are non-trivial, then the client is not a very robust one.

If an application developer is willing to abandon a solution as generic and powerful as this modulei is because the details of paginated requests can't be abstracted over, then I'm not sure we are going to be able to make useful progress in this issue queue.

I realize that was all very blunt, but I'd like to extend a branch, if you would like to write such a JS client library for this module, I would absolutely be willing to work with you to implement pagination in a way that the details of it are hidden from end-developers. I'm sure @e0ipso would be too.

Let's work together on building something rather than arguing over a simple implementation detail.

Here is an example of the a well-respected API choosing a lower default and a limit that is only twice as high.

https://developer.github.com/v3/#pagination

There are some excellent resources there that discuss pagination.

Perhaps we can work together to provide snippets of JavaScript that can illustrate how to implement pagination well.

Hi @gabesullice, thanks for chiming in! I really appreciate your input.

I do however have to strongly disagree with much of what you said... so please don't take this as me being aggressive, I'd just like to point out my view of things which as a front end developer is drastically different than yours.

we have managed to keep it to ZERO configuration with sane defaults

I don't think this statement is true at all. You support sorting and filtering correct? Pagination options are no different than that level of configuration. In the json api spec ( http://jsonapi.org/format/#fetching-pagination ) , it specifically allows for a page[size] value. I have no issues with a default size of 50, but I don't see the logic in preventing a developer from increasing the page size as specified in the docs.

If it's a hefty node type, this could easily cause PHP to run out of memory and crash

Sure it could. And if it's a thin node type, it could handle 20,000 easily. In my give.wvu.edu example I listed earlier, I load 300 nodes in 200ms including transport. I could do that all day. You can't presume to know the use case for the Json API module, and I would argue it should be flexible enough to allow the developer to make his own decisions... *at least* to the point of following the json api documentation.

Truly, this is a question of application developer competency (I'm sorry to put it so bluntly)

I'll try not to take offense at this :P

Put simply, the debate is not about the "right" limit or even if a limit should exist at. But simply about best-practices. If one is not writing a client in such a way that paginated results are non-trivial, then the client is not a very robust one.

Best practices for a drupal site can be different from best practices for a progressive web app. In a web app the whole page loads instantly, and you can have a little tiny spinner while your data loads. In some use cases (such as when you're going to be on a single page for hours working with the data) an extra 2 or 3 seconds of load time on that spinner in the beginning is preferable to constantly making calls back to the server for data.

Here is an example of the a well-respected API choosing a lower default and a limit that is only twice as high.

https://developer.github.com/v3/#pagination

This is where I think the real crux of the problem is. You're blurring the lines between an API and a tool used to serialize data. That github link you send me is an API and the developers of that API made a conscious decision based on use cases for their API, usage, and server capabilities that 100 results is effective for their system. The actual output of github's api is not limited to 100 because of their CMS software. That would be silly, it's limited to 100 because they were able to determine that's what makes the most sense for github and they set it there.

JSON API module is a tool to help developers build APIs. By hard coding a limit of 50, you're preventing everyone that wants to use the module from making those same business decisions that github did. In fact, if github wanted to switch it's CMS to drupal, they would not be able to use the JSON API module, or you would force them to double their requests (2x50 results) even though they've already determined 100 makes the most sense for them. I'm sorry, that's just silly.

I recognize that implementing pagination is not easy.

~~~

I realize that was all very blunt, but I'd like to extend a branch, if you would like to write such a JS client library for this module, I would absolutely be willing to work with you to implement pagination in a way that the details of it are hidden from end-developers. I'm sure @e0ipso would be too.

~~~

Perhaps we can work together to provide snippets of JavaScript that can illustrate how to implement pagination well.

Again I'll try not to take offense here. Pagination isn't difficult in concept or in execution. I think this is more a traditional web page vs. progressive web app mentality disconnect we have here. In app development pagination doesn't always make sense. Sometimes it does. Usually people prefer an extra few seconds up front for better performance, but again I feel like it's not the JSON API module's place to determine that for a developer, it's should be up to the developer to determine what makes the most sense for his program.

I'm not opposed to contributing, but you're asking me to build an entire JS client library to work around a hard coded 50 result limit you put in because it made sense to your particular use case. That is also silly.

Here's a list of already built client libraries that should work out of the box with anything using the JSON API spec, but are broken because of this decision:

http://jsonapi.org/implementations/

So if I need more than 50 results per request, instantly all of these wonderful tools are broken for me. There's even one specifically for react / redux ( https://github.com/dixieio/redux-json-api ) which looks fantastic, but can't use it because the JSON API maintainers decided they wanted to hard code in a limit that's out of spec with the JSON API spec.

That's silly. This whole discussion is silly.

I really didn't want this to turn into a big thing guys sorry, I'm 100% willing to just walk away and let you guys do your thing. I really do appreciate your contributions to drupal even if I disagree with some design decisions.

Filtering, sorting, and pagination are not configuration. What I meant was module configuration pages, like what entities should and shouldn't be exposed by the API or what path pattern we use for our endpoints. Or the maximum allowable entities in a request.

We must ship with some maximum.

Here's why: If you allow a client to specify any limit, it is a security flaw in the module. You as the frontend developer might choose a sane limit, but any malicious user can simply open up their console and send a server-crashing request by upping the page limit. If you would like me to further explain what I mean, I'd be happy to.

So, if we take it as given that we must ship with some maximum (again, if you want to hop on slack or something I can try to better explain why this is a fact).

We must have a maximum. You still may want to change what that maximum is on the server side (because changing it on the client side is a security vulnerability) to better accommodate your particular application needs. What can we do?

1. Make a configuration page where it can be changed
2. Let users override our code using established conventions

As I said, this module had zero configuration. We have no config page. We've had lots of positive feedback about this decision. However, we see certain things that we imagine we could make configurable down the road, like mapping field_my_field to myFieldAlias. But, at this time, the module doesn't have configuration options (this is why I postponed the issue earlier).

That leaves option two, which you said is unacceptable.

What are we to do?

We chose a conservative default so we would hopefully not break for unexpected reasons. We implemented pagination according to a well-established specification.

If those libraries break because of pagination, those libraries are broken. If they cannot follow page links to fulfill a request for 200 items when the page limit is 50 and the spec has been implemented correctly, those libraries are incomplete. Again, happy to further explain why.

Let's not call this debate silly. It trivializes the issues we're discussing on both sides of the argument.

I truly believe this all boils down to a different understanding of terms. Like what we mean by configuration and what we mean by pagination or configuration.

I will be on slack soon and then for the rest of the day, we can try to come to a better understanding of one another's terminology.

I really want this issue to be resolved here so that those landing on the issue have a good answer for their needs. I sincerely appreciate your zeal, it will help us provide the best solution that we can for the greatest number of people.

@gabesullice whats wrong with the third option of using page[size] as per the json api specs?

It wouldn't require a configuration page, and wouldn't allow a client to override?

/users?page[size]={page_size}
{page_size} = the number of resources to be returned per page
e.g. /users?page[size]=25 will return 25 user resources on a single page

@endorn do you use Slack or IRC? I feel like we've addressed the reason that that is not a good solution a few times; something must be lost in translation. Either I am not understanding your idea, or vice versa.

@gabesullice yeah I have slack

@endorn go ahead and join the Drupal Slack (https://www.drupal.org/slack) and you can find me on there. We can figure out where we're misunderstanding one another. As an outcome, I'd like to end up with a new issue to fix a problem or a page of documentation that we can work on to help clarify the design decision. My username is gabesullice.

@endorn, @gabesullice did you come guys to any conclusion in the end?

I read the whole conversation and can tell that both parties are right: there is a good reason not to allow the frontend to decide how many items should be loaded per page by default. However, there should be an easy way for the backend to allow the frontend to load more than default. Suggestion in #7 is nice, but it's too complex for non-professional Drupal devs.

Considering the fact that JSON API is zero configuration module (which is great in general, but sometimes leads to tough decisions), I would propose to make the page limit value configurable in JSON API Extras. So if backend devs know what they are doing and they explicitly say "Yes, I still want to allow frontend to load 5k entities at once" then they can do this by setting the appropriate configuration in the UI or settings.php. Will this work for everyone?

@Spleshka that sounds like a nice feature compromise. Perhaps that setting should be entity type/bundle specific? @e0ipso will have to answer viability questions. I'm not familiar with the jsonapi_extras code base.

@Spleshka I'd love to review that patch!

@gabesullice Sounds good to me.

@e0ipso that's a tricky way to request for a patch :P Createad a follow-up in JSON API Extras #2884292: Make max value of page[limit] configurable per entity/bundle, assigned to myself.

@Spleshka hehehe >:-D

@spleshka per entity type configuration would be annoying but acceptable which is probably exactly where we'd want to be with this. Make it possible, but not easy so that you only do it if you are 110% sure you need to.

@endorn The only reason I did not implement auto pagination in the training was because it is a basic training, not a production ready application. Please do not use that as mark against this module. If anything, it was my job as the trainer to understand that limitation and create guards around the training so that it wouldn't be an issue.

I take issue with your approach here and labeling it as "passion" or "zeal" doesn't excuse it. Your tone is reminiscent of many others I've seen in OSS. One aspect of a library doesn't work for your use case so you label the decision as "silly" and say you'll have to "walk away" and use another project. It's not becoming and is not helpful.

If the projects linked at http://jsonapi.org/implementations/ cannot handle automatic pagination, then they are broken - not the inverse. This is a very common and no-brainer limitation of an API and could be easily solved for in one of these client side libraries. PRing those projects to improve their pagination strategy rather than harping on this module would be more constructive than complaining that you can't plug and play this module. I'm certain that other servers that implement JSON API would and should do so with a default limit to their query size parameter. Improving existing client-side libraries to handle this case would be much more valuable to the entire web development community. You could start by re-opening this issue: https://github.com/stonecircle/redux-json-api/issues/126 which was written off as an "API implementation issue" and perhaps providing a PR that implements your use case. If that is not an option I would encourage you to check out https://www.drupal.org/project/subrequests for bundling many requests. If that is not an option, then ask a backend developer to swap out the service as mentioned.

Reach out to me on Drupal Slack @infiniteluke if you want to discuss this further.

Has there been any documentation as of yet around extending your own service to overwrite this? I get the basics of it, but if there is something out there, I would love to take a look before possibly setting off down this path.

Auto-pagination? Subrequests? Parallel paginated requests? I just want to be able to pull down ~150 names to populate a dropdown from a controlled, developer-only internal source with whitelisting. Why not keep the module clean, on-spec, and free from bias and assume that developers know their users best and let them assume responsibility for their own sites? Result limiting of any kind could be moved to JSON API Extras for those who intentionally choose it, still keeping everything configuration free. Otherwise, it's a barrier to adoption; many teams just don't have the resourcing, skill, or time for the aforementioned sophisticated workarounds. It may be a hard sell in such simple use cases; most may just opt to stick with JsonResponse objects or consuming manually created JSON View endpoints.

That said, I did manage to spend the day figuring out how to get around this imposed requirement by extending the service by:
1. Extending 'jsonapi.entity_resource' (services.yml) with my own CustomEntityResource class (extends Drupal\jsonapi\Controller\EntityResource).
2. Overriding the getJsonApiParams method, swapping out OffsetPage for my own CustomOffsetPage class.
3. Overriding const SIZE_MAX in CustomOffsetPage to set it to 500 (or whatever).

I don't know anything about contributing modules to Drupal (only been working with it < 2 years), but I uploaded some module code to https://github.com/manafire/jsonapi_custom_limit for anyone interested.

I would like to thank the maintainers and contributors for their considerable time and effort on this impressive module. I hope to continue to keep using it, but wanted to share the difficulties with the page limit as the added workaround complexities may ultimately prove a dealbreaker for our team.

I would like to thank the maintainers and contributors for their considerable time and effort on this impressive module

Thanks for the kind words ❤️

Thank you, manafire for the example.

We have a similar use case and I used your example to remove the limit. The code I used could also be modified to replace the limit with a different value. I tried to do as little copy & pasting as possible. The other major change is to use a Service Provider to override the service rather than the service definition. I think this is more reliable since it's guaranteed to run after the YML files are parsed.

I also changed the names of the files removing "Custom" and using "use ... as ..." patterns. That last bit is a preference thing, but I wanted to highlight it in case it was helpful to someone else.

Our file structure
example_jsonapi_limit.info.yml
example_jsonapi_limit.services.yml
src/ExampleJsonapiLimitServiceProvider.php
src/Controller/EntityResource.php
src/Query/OffsetPage.php

I'll paste in the relevant bits. Keep in mind that I've tried to sanitize the project name so this might not work 100% copy and paste.

example_jsonapi_limit.services.yml
This defines a new service and doesn't try to overwrite the old one.

services:
  example_jsonapi_limit.entity_resource:
    class: Drupal\example_jsonapi_limit\Controller\EntityResource
    arguments:
      - "@entity_type.manager"
      - "@entity_field.manager"
      - "@jsonapi.resource_type.repository"
      - "@renderer"
      - "@entity.repository"
      - "@jsonapi.include_resolver"
      - "@jsonapi.entity_access_checker"
      - "@jsonapi.field_resolver"
      - "@jsonapi.serializer"
      - "@datetime.time"
      - "@current_user"

src/ExampleJsonapiLimitServiceProvider.php

/**
 * @file
 * Contains \Drupal\example_jsonapi_limit\ExampleJsonapiLimitServiceProvider.
 */

namespace Drupal\example_jsonapi_limit;

use Drupal\Core\DependencyInjection\ServiceModifierInterface;
use Drupal\Core\DependencyInjection\ContainerBuilder;

/**
 * Override the service used by jsonapi to build the entity resource.
 *
 * This way we can remove the limit on the number of items to return.
 */
class ExampleJsonapiLimitServiceProvider implements ServiceModifierInterface {

  /**
   * {@inheritdoc}
   */
  public function alter(ContainerBuilder $container) {
    $container->setDefinition(
      'jsonapi.entity_resource',
      $container->getDefinition('example_jsonapi_limit.entity_resource')
    );
  }
}

src/Controller/EntityResource.php

namespace Drupal\example_jsonapi_limit\Controller;

use Drupal\jsonapi\Controller\EntityResource as DefaultEntityResource;
use Drupal\jsonapi\ResourceType\ResourceType;
use Drupal\example_jsonapi_limit\Query\OffsetPage as OffsetPage;
use Symfony\Component\HttpFoundation\Request;

/**
 *
 * Overridden and extended from jsonapi module to get around the 50 results limit
 *
 */
class EntityResource extends DefaultEntityResource {

  /**
   * Extracts JSON:API query parameters from the request.
   *
   * @param \Symfony\Component\HttpFoundation\Request $request
   *   The request object.
   * @param \Drupal\jsonapi\ResourceType\ResourceType $resource_type
   *   The JSON:API resource type.
   *
   * @return array
   *   An array of JSON:API parameters like `sort` and `filter`.
   */
  protected function getJsonApiParams(Request $request, ResourceType $resource_type) {
    $params = parent::getJsonApiParams($request, $resource_type);

    // Use our overridden methods to replace the page parameters if they are
    // present.
    if ($request->query->has('page')) {
      $params[OffsetPage::KEY_NAME] = OffsetPage::createFromQueryParameter($request->query->get('page'));
    }

    return $params;
  }
}

src/Query/OffsetPage.php

namespace Drupal\example_jsonapi_limit\Query;

use Drupal\jsonapi\Query\OffsetPage as DefaultOffsetPage;

/**
 * Value object for containing the requested offset and page parameters.
 *
 * Overrides OffsetPage from the jsonapi module to get around the 50 results limit.
 */
class OffsetPage extends DefaultOffsetPage {

  /**
   * Override this method to remove checking against a max size.
   *
   * @param mixed $parameter
   *   The `page` query parameter from the Symfony request object.
   *
   * @return static
   *   An OffsetPage object with defaults.
   */
  public static function createFromQueryParameter($parameter) {
    $offset_page = parent::createFromQueryParameter($parameter);

    // Rebuild the parameter if the set size is larger than the maximum.
    if ($parameter[static::SIZE_KEY] > static::SIZE_MAX) {
      // Do not overwrite the size option for parameter.
      $expanded = $parameter + [
        static::OFFSET_KEY => static::DEFAULT_OFFSET
      ];

      $offset_page = new static($expanded[static::OFFSET_KEY], $expanded[static::SIZE_KEY]);
    }

    return $offset_page;
  }

}

I know I'm visiting an old issue - at least now that JSON:API is in core - but I would like to offer another approach to this issue which I used in a project of mine. The problem with the parallel requests is that if you really need to obtain 300 entities at once (think of a calendar displaying events), then you would have to get the first 50 entities in a single request, get the `count` that they are actually 300 in total, divide it with 50 which is the max size set as a constant, then do the proper parallel requests and merge the results before proceeding with the rendering.

I don't criticize the decision to make it a constant value to be overriden only by a new service (like the one set by jsonapi_page_limit module). There are good reasons for it and in most cases they are really good reasons!

Just in case it is required, the approach I would like to offer is to use the patch I attach. I find my solution better than using the jsonapi_page_limit module since
(a) you don't require an extra module,
(b) it is a really simple patch that's not expected to break any time soon and
(c) it's completely managed by composer and common patch techniques.

This patch is for the Drupal 8+ versions having the JSON:API in core and you can add it to your project like this:

1. Store it in a local folder of your project, outside your [web-root] folder (in my case it's called patches).
2. Edit the file and set the upper limit to the upper limit you like.
3. Update your composer.json file with the following:

"extra": {
  ...
  "patches": {
      "drupal/core": {
          "Increase JSON:API max limit": "patches/increase_json_api_max_limit.patch"
      }
  }
  ...
}

PS: If you find it helpful, you could consider linking this response to the documentation.

Thanks for the contribution @vensires! I am sure this will be helpful to other people as well.

@vensires Thank you for your patch! I have been using it for quite some time. But it doesn't apply anymore.

@e0ipso I am still wondering if we can't make this happen somehow...

Lukas, this will not happen. At least not in this project, since this has been already added to core. You could try to move the issue to Drupal core and take it from here.

@lukas-von-blarer I just checked Drupal 8.9.14 and it still applies fine. Maybe it doesn't apply to the D9 version?

We use the jsonapi just for server side requests (node.js / next.js). We need to catch therefore a really massive data export from the json api. Is the only solution in the moment to stick to custom json responses via custom modules instead of the jsonApi? The jsonApi Extras / Default module shows a limit override field. Is this not usable anylonger in d9+ ?

Thanks