Large request bodies results in PHP fatal "out of memory" error while decoding JSON

What is the display_errors PHP setting?

@cilefen this is encountered when display_errors is set to true or false.

When a client would ask me: We need to be protected against attackers, I would start telling them: That is not that easy. There are many vectors:typical problems are for example DDos/cache bypassing attacks etc. but then far along the line there is this. Sure we could try to fix it in PHP, but do we really believe this is a good place to fix it? I would argue that filtering out on the server level is the only proper way to fix this particular problem.

I agree that there is only so much we can do here. Drupal can never protect against all attack vectors here. I do see two simple things we can do to address the most extreme cases though:

1. automatically decline requests whose bodies contain contents larger than the PHP memory limit
2. introduce a max_request_body_size setting in rest.settings, set to e.g. 1 megabyte by default (if we do this, the above would not be necessary of course)

It'd help to know whether in @malik.kotob's case the request body was just "big" (say half a megabyte) or "impossibly enormous" (say, 20 megabytes). Can you provide that information, @malik.kotob?

I agree in that we can't seek out all attack vectors, but if one presents itself I don't see any reason not to contribute a fix that can help someone else out in the community. With that being said if this isn't the right place to do it then I definitely wouldn't argue in favor of it.

@Wim Leers:
1. This would help in some requests, but wouldn't prevent the error when the body is smaller than the memory limit, but larger than what is available in memory. Maybe you're not suggesting that it would, but just wanted to point that out in case.
2. I see this as a very viable solution. In our case the request body was enormous, about 50MB. We are, however, accepting requests that are ~30MB without issue, which is a valid use case for us. The trouble with only filtering things out at the server level is that we may not want to limit an admin, for example, from uploading a 50MB file. We truly want to impose this limit only on our custom endpoint.

This would help in some requests, but wouldn't prevent the error when the body is smaller than the memory limit, but larger than what is available in memory. Maybe you're not suggesting that it would, but just wanted to point that out in case.

The problem is that it's impossible to determine how much memory decoding a payload will consume…

We are, however, accepting requests that are ~30MB without issue, which is a valid use case for us.

This would most definitely be MUCH larger than what I would consider a sane default. 30 MB request bodies, wow, what?! Does this include base64-encoded files?

We truly want to impose this limit only on our custom endpoint.

If that's the case, that's trivial?

function post(…, Request $request) {
  if (Unicode::strlen($request->getContent()) > 30 * 1024 * 1024) {
    throw new HttpException(507);
  }
  …

The problem is that it's impossible to determine how much memory decoding a payload will consume…

Pardon my ignorance, but is there a way to make a conservative estimate? If not, I think the one potential solution you outlined could work well:

introduce a max_request_body_size setting in rest.settings, set to e.g. 1 megabyte by default

This would most definitely be MUCH larger than what I would consider a sane default. 30 MB request bodies, wow, what?! Does this include base64-encoded files?

It does, unfortunately, and the problem with using the above snippet, which I think you're saying could be added to our custom resource endpoint that extends ResourceBase, is that the memory issue is hit before our post implementation is (in rest's handle method in RequestHandler.php). That's why my thought was that if it's not possible to conservatively calculate how much memory the decoding will take,
introducing the max_request_body_size setting would give us something to work with upstream.

is there a way to make a conservative estimate?

No, because that'd require Drupal to estimate the algorithmic complexity/memory consumption of external libraries…

the memory issue is hit before our post implementation is

Your number one problem is that you're using a base64-encoded payload. That's a big red flag.

While not pretty, you could handle that by modifying the _controller that is specified in \Drupal\rest\Plugin\ResourceBase::getBaseRoute(), because you can override that method. That way the complexity remains tied to your rather unfortunate (but required, I'm sure!) @RestResource plugin, rather than making the core REST module more complex.

I hope that makes sense to you as well!

Thanks for the explanation and tip!