Try to use HTTP trailers or a real streaming response in Big Pipe.

https://github.com/symfony/symfony/issues/46449 was closed ~30 mins ago: the upstream change that caused this regression was reverted.

But @stof makes an interesting point at https://github.com/symfony/symfony/issues/46449#issuecomment-1136082446:

@catch56 if you send content in a special way, BigPipeResponse should probably set `false` as the buffered content of the response instead of setting the unprocessed content.

That seems reasonable too. \Symfony\Component\HttpFoundation\StreamedResponse::setContent() does something similar.

This patch does that.

And

> (you could also extend from StreamedResponse if that'd make sense btw)

Unfortunately we can't, because `class BigPipeResponse extends HtmlResponse` and `class HtmlResponse extends Response implements CacheableResponseInterface, AttachmentsInterface`.

> Stream

adds to that.

I was going to respond that unfortunately we can't, because class BigPipeResponse extends HtmlResponse and class HtmlResponse extends Response implements CacheableResponseInterface, AttachmentsInterface.

But … HtmlResponse barely does anything. So we can absolutely change this. Nope, we can't, because we have lots of code doing $response instanceof HtmlResponse checks … 😬

we have lots of code doing $response instanceof HtmlResponse checks … 😬

I bet we can touch all this places

I suppose we should do this before Drupal 10?

I think as it's task it doable in 9.5 too

This is important also because any post-response processing (e.g., destructable services) keeps the client "spinner" spinning when the server uses mod_php, even though all the content has been sent. See #3295790-69: Post-response task running (destructable services) are actually blocking; add test coverage and warn for common misconfiguration.

Idea: if Symfony actually added support for HTTP trailers, then we could change BigPipe to rely on that instead — we wouldn't have to mess with Response::$content anymore!

Still zero interest in this upstream — maybe you and I could collaborate to figure out how to bring that to Symfony?

EDIT: and that would in principle also allow us to bring https://www.drupal.org/project/big_pipe_sessionless to Drupal core — see the description on that page which also mentions HTTP trailers 🤓

Quoting MDN:

The Trailer response header allows the sender to include additional fields at the end of chunked messages in order to supply metadata that might be dynamically generated while the message body is sent, such as a message integrity check, digital signature, or post-processing status.

IOW: it still requires chunked/streamed responses … but it just would fit better with Symfony's desire/intent to disallow manipulating Response::$content.

That's it.

No HTTP-level reason. Only a Symfony-level reason.

Found this after looking into BigPipe behind Amazon CloudFront, where I don't appear to be seeing the content streamed correctly - instead the response is delivered all in one go.

CloudFront docs say:

CloudFront supports only the chunked value of the Transfer-Encoding header. If your origin returns Transfer-Encoding: chunked, CloudFront returns the object to the client as the object is received at the edge location, and caches the object in chunked format for subsequent requests.

BigPipe.php says

    // BigPipe sends responses using "Transfer-Encoding: chunked".

but it appears this is not actually true?

As far as I've seen, Big Pipe does send chunked? All you have to do is print something and then call flush(); and the Transfer-Encoding: chunked header should be added automatically. HTTP/2 or HTTP/3 don't support it though, so you might not literally see a chunked transfer, depending on the stack.

re: compatible with HTTP/2, I'd hope that proxies convert HTTP/1.1 chunked transfer to HTTP/2 (or HTTP/3) data frames and pass them along, without excessive buffering, but not something I've analyzed