[upstream] BigPipe breaks with symfony/http-foundation 6.1

Objects Responses in the Mirror Big Pipe Are Closer Bigger Than They Appear: Addition to symfony/http-foundation/Response::send()

BigPipeResponses are getting a Content-Length header that's based on the size before the replacement magic happened, which leads Responses being cut-off before their "true" end.

Attached patch works(1) (Down from 27 to 22 failures (EDIT: Yes, 22 and boo! to random JS test failures)), doesn't break the current SF6.0 setup(2), but feels dirty and more of a workaround than actually solving anything.

Since the Content-Length header is set in symfony/http-foundation/Response::send() and that's being at the very end of the chain, I think we could do something like wrapping it in a \Drupal\Core\DrupalKernel::send function where we strip the Content-Length header if the Response is a \Drupal\big_pipe\Render\BigPipeResponse, and swap the $response->send(); here with our custom function.

Or maybe this is something that should be addressed upstream in symfony/http-foundation.

Anyway: I'll happily leave decisions like that to the Bigger Brains around here. Putting this on NR to attract those Brains.

But do we need to do a lot of work to use Transfer-Encoding: chunked or is it just a case of setting the header? I guess there's an easy way to find out so uploading a patch.

I think I tried this (or something similar) locally and that failed tests as well.
Anyway: Locally and _think_ in one sentence, so we need a proper patch in here, thanks for that.

If we can't use the proper header, then #6 actually seems cleaner than doing this in DrupalKernel - it shouldn't need to know about BigPipe

Agreed.

Feels like this is worth an upstream issue to at least tell them what happened.

Agreed, I think we''ll wait on #3276186-9: [upstream] BigPipe breaks with symfony/http-foundation 6.1 return from The Land of TestBot before doing so?

@catch So (what seems to be) the correct way doesn't work.

I think we need to open an upstream issue, and I think if you did it, the issue would have a bit more gravitas (yes, I picked up a new word and want to use it more in conversation...). Also I think your Big Brain would jolt it down a bit more to the point than my incoherent ramblings.

Could you open it?

Maybe it's related to PHP https://github.com/php/php-src/pull/8353

Rrrrrandom JS testfailure. Back to NR

Symfony 6.1 is out.

Looks like an upstream patch may take a while, other users reported similar issues, the Symfony team is looking for a test app that reproduces the issue. Maybe we can get this in to get going, the @todo is there, will need a follow up to apply the upstream fix once available.

PHPStan :)

Also, maybe it's better open a followup in our queue and reference that in the @todo

Thanks, RTBC. See also https://drupal.slack.com/archives/C014CT1CN1M/p1653913440894539

+++ b/core/modules/big_pipe/src/Render/BigPipeResponse.php
@@ -105,6 +105,27 @@ public function setBigPipeService(BigPipe $big_pipe) {
+    elseif (!\in_array(\PHP_SAPI, ['cli', 'phpdbg'], TRUE)) {
+      static::closeOutputBuffers(0, TRUE);

interesting how it will affect swoole

I see, thanks! +1 rtbc

Wow, what a huge BC break! :O

The issue we have is that big_pipe implements 'chunking' without actually using Transfer-Encoding: chunked (Drupal\big_pipe\BigPipe even claims that it uses the header).

Indeed, it does, and I can observe it being set locally, at least according to Chrome. Just stepped through it with xdebug and indeed, $response as received by public function sendContent(BigPipeResponse $response) does not have that header set!

print_r($response->headers->all(), TRUE)

gives

Array
(
    [content-type] => Array
        (
            [0] => text/html; charset=UTF-8
        )

    [cache-control] => Array
        (
            [0] => must-revalidate, no-cache, private
        )

    [date] => Array
        (
            [0] => Tue, 31 May 2022 15:50:42 GMT
        )

    [x-drupal-dynamic-cache] => Array
        (
            [0] => HIT
        )

    [link] => Array
        (
            [0] => <http://d8.test/>; rel="canonical", <http://d8.test/>; rel="shortlink"
            [1] => <http://d8.test/>; rel="canonical", <http://d8.test/>; rel="shortlink"
        )

    [surrogate-control] => Array
        (
            [0] => no-store, content="BigPipe/1.0"
        )

    [x-accel-buffering] => Array
        (
            [0] => no
        )

    [x-ua-compatible] => Array
        (
            [0] => IE=edge
        )

    [content-language] => Array
        (
            [0] => en
        )

    [x-content-type-options] => Array
        (
            [0] => nosniff
        )

    [x-frame-options] => Array
        (
            [0] => SAMEORIGIN
        )

    [permissions-policy] => Array
        (
            [0] => interest-cohort=()
        )

    [x-drupal-cache-tags] => Array
        (
            [0] => 4xx-response block_view config:block.block.olivero_account_menu config:block.block.olivero_breadcrumbs config:block.block.olivero_content config:block.block.olivero_help config:block.block.olivero_main_menu config:block.block.olivero_messages config:block.block.olivero_page_title config:block.block.olivero_powered config:block.block.olivero_primary_admin_actions config:block.block.olivero_primary_local_tasks config:block.block.olivero_search_form_narrow config:block.block.olivero_search_form_wide config:block.block.olivero_secondary_local_tasks config:block.block.olivero_site_branding config:block.block.olivero_syndicate config:block_list config:search.settings config:system.menu.account config:system.menu.admin config:system.menu.main config:system.site config:system.theme http_response local_task rendered
        )

    [x-drupal-cache-contexts] => Array
        (
            [0] => cookies:big_pipe_nojs languages:language_interface route session.exists theme url.path.is_front url.path.parent url.query_args:_wrapper_format user.permissions user.roles:anonymous user.roles:authenticated
        )

    [x-drupal-cache-max-age] => Array
        (
            [0] => 0 (Uncacheable)
        )

    [expires] => Array
        (
            [0] => Sun, 19 Nov 1978 05:00:00 GMT
        )

    [x-generator] => Array
        (
            [0] => Drupal 9 (https://www.drupal.org)
        )

)

There's a further issue though that http/2 doesn't support Transfer-Encoding: chunked although I'm not clear what this means for responses that send it.

🤯 Source?

+++ b/core/modules/big_pipe/src/Render/BigPipeResponse.php
@@ -105,6 +105,27 @@ public function setBigPipeService(BigPipe $big_pipe) {
+    // cspell:ignore litespeed phpdbg
+    if (\function_exists('fastcgi_finish_request')) {
+      fastcgi_finish_request();
+    }
+    elseif (\function_exists('litespeed_finish_request')) {
+      litespeed_finish_request();
+    }
+    elseif (!\in_array(\PHP_SAPI, ['cli', 'phpdbg'], TRUE)) {
+      static::closeOutputBuffers(0, TRUE);
+    }

I don't like that we're repeating all this … but fortunately https://github.com/symfony/symfony/issues/46449 was fixed 23 minutes ago, by reverting the upstream change that caused this 👍

I wonder if it makes more sense to wait for a new Symfony 6.1 patch release? I'm fine with moving ahead here and reverting all of this in #3283145: Try to use HTTP trailers or a real streaming response in Big Pipe. of course 😄

AFAICT: This was resolved with SF 6.1.1, so I'm closing this as outdated.