Problem/Motivation

content-length is set in a Response kernel event. But http_middleware services come later. Page cache is one example of a middleware. In the case of it, it doesn't alter the response. But until now, nothing stopped a middleware from altering the response. This messes up sites that are using any middlewares that do just that.

Steps to reproduce

Proposed resolution

Instead of setting content length in a kernel event, set it later in a http middleware.

Remaining tasks

User interface changes

API changes

Change notice https://www.drupal.org/node/3298551 should be updated explaining some of the implications related to adding the content length to HTTP headers.

Data model changes

Release notes snippet

CommentFileSizeAuthor
#4 3410022.patch5.61 KBheddn

Issue fork drupal-3410022

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

heddn created an issue. See original summary.

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented
Title: Regression from #3295790 content-length header earlier then expected » Regression from #3295790 content-length header set earlier then expected
heddn’s picture

heddn
Primary language English
Location Nicaragua
commented
Issue summary: View changes

I'm able to confirm that this is a regression. If I remove the content length header from HtmlResponseBigPipeSubscriber and FinishResponseSubscriber, then everything starts working again. This breaks https://www.drupal.org/project/tailwindcss_utility, which has a http_middleware service.

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented
Status: Active » Needs review
StatusFileSize
new 3410022.patch5.61 KB

While looking at things, I wonder if we need to also alter the Cors middleware to a lower weight. It would also adjust the content length. For now, leaving alone. This just gets the conversation started.

smustgrave’s picture

smustgrave commented
Issue tags: +Needs tests, +Needs issue summary update, +Needs change record

Adding some needed tags

bradjones1’s picture

bradjones1
Primary language English
commented
Status: Needs review » Needs work

Should also be an MR at this point.

heddn changed the visibility of the branch 3410022-regression-from-3295790 to hidden.

heddn changed the visibility of the branch 3410022-regression-from-3295790 to hidden.

heddn changed the visibility of the branch 3410022-regression-from-3295790 to active.

heddn opened merge request !5920

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented
Issue summary: View changes
Status: Needs work » Needs review

Converted to an MR and a failing test added.

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented
Issue tags: -Needs tests, -Needs issue summary update, -Needs change record +Needs change record updates

Updating tags

catch’s picture

catch
he/him
Primary language English
commented

This will conflict with #3396559: Only set content-length header in specific situations but it might fix any remaining issues related to the change.

larowlan’s picture

larowlan
Location 🇦🇺🏝.au GMT+10
commented
Issue tags: +Patch release target
heddn’s picture

heddn
Primary language English
Location Nicaragua
commented

I'm happy to rebase this if the other one goes in first.

larowlan’s picture

larowlan
Location 🇦🇺🏝.au GMT+10
commented

Reworked after #3396559: Only set content-length header in specific situations

mstrelan’s picture

mstrelan commented
Title: Regression from #3295790 content-length header set earlier then expected » Regression from #3295790 content-length header set earlier than expected
catch’s picture

catch
he/him
Primary language English
commented
Status: Needs review » Reviewed & tested by the community

We do set a X-Accel-Buffering header in BigPipeResponse, perhaps we could check for that instead?

This is tempting, but I like the separate middleware too, seems a bit more explicit.

Can't see anything to complain about here, so moving to RTBC.

longwave’s picture

longwave
he/him
Primary language English
Location UK
commented

Three tests failed, two of them in BigPipe, don't recognise them as randoms so triggered a retest but if they fail again there are more problems to fix here.

longwave’s picture

longwave
he/him
Primary language English
Location UK
commented
Status: Reviewed & tested by the community » Needs work

Yup, AssetOptimizationTestUmami::testAssetAggregation and two of the BigPipe tests fail consistently.

catch’s picture

catch
he/him
Primary language English
commented
Status: Needs work » Needs review

Pushed a commit for that - priority is highest first, it was using them like weights.

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented
Status: Needs review » Reviewed & tested by the community

For me, this is RTBC again if the tests pass green.

  • longwave committed 257ca110 on 10.2.x 
    Issue #3410022 by heddn, larowlan, catch: Regression from #3295790...

  • longwave committed 8f087227 on 11.x 
    Issue #3410022 by heddn, larowlan, catch: Regression from #3295790...
longwave’s picture

longwave
he/him
Primary language English
Location UK
commented
Version: 11.x-dev » 10.2.x-dev
Status: Reviewed & tested by the community » Fixed
Issue tags: -Needs change record updates

Committed and pushed 8f0872276e to 11.x and 257ca110a6 to 10.2.x. Thanks!

Also updated the change record at https://www.drupal.org/node/3298551 to reflect the new way of doing this, would appreciate it if someone can double check my work there.

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented

change record looks good to me.

andypost’s picture

andypost
he/him
Primary language Russian
commented

Please close the MR

heddn closed merge request !5920

godotislate’s picture

godotislate
he/him
commented

Not sure whether to post a follow up against Drupal core or drush, but after this commit, I'm seeing exceptions being thrown when I run drush updb. Database updates via web UI work just fine. Drush version is 12.4.3.0.

Steps to reproduce:

  1. Install latest Drupal 10.2.x with standard profile
  2. Run drush updb
  3. Observe exceptions thrown in console
Fatal error: Uncaught Symfony\Component\VarExporter\Exception\LogicException: Cannot generate lazy proxy: class "Drupal\big_pipe\StackMiddleware\ContentLength" is final. in /var/
  www/html/vendor/symfony/var-exporter/ProxyHelper.php:92
  Stack trace:
  #0 /var/www/html/vendor/symfony/dependency-injection/LazyProxy/PhpDumper/LazyServiceDumper.php(136): Symfony\Component\VarExporter\ProxyHelper::generateLazyProxy(Object(Reflectio
  nClass), Array)
  #1 /var/www/html/vendor/symfony/dependency-injection/LazyProxy/Instantiator/LazyServiceInstantiator.php(33): Symfony\Component\DependencyInjection\LazyProxy\PhpDumper\LazyService
  Dumper->getProxyCode(Object(Symfony\Component\DependencyInjection\Definition), 'http_middleware...')
  #2 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1086): Symfony\Component\DependencyInjection\LazyProxy\Instantiator\LazyServiceInstantiator->instantiate
  Proxy(Object(Drupal\Core\DependencyInjection\ContainerBuilder), Object(Symfony\Component\DependencyInjection\Definition), 'http_middleware...', Object(Closure))
  #3 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\Dep
  endencyInjection\Definition), Array, true, 'http_middleware...')
  #4 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tru
  e)
  #5 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Componen
  t\DependencyInjection\Reference), Array, true)
  #6 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #7 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\Dep
  endencyInjection\Definition), Array, true, 'http_middleware...')
  #8 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tru
  e)
  #9 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Componen
  t\DependencyInjection\Reference), Array, true)
  #10 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #11 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\De
  pendencyInjection\Definition), Array, true, 'http_middleware...')
  #12 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tr
  ue)
  #13 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Compone
  nt\DependencyInjection\Reference), Array, true)
  #14 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #15 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\De
  pendencyInjection\Definition), Array, true, 'http_middleware...')
  #16 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tr
  ue)
  #17 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Compone
  nt\DependencyInjection\Reference), Array, true)
  #18 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #19 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\De
  pendencyInjection\Definition), Array, true, 'http_middleware...')
  #20 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tr
  ue)
  #21 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Compone
  nt\DependencyInjection\Reference), Array, true)
  #22 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #23 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\De
  pendencyInjection\Definition), Array, true, 'http_kernel')
  #24 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(531): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_kernel', 1)
  #25 /var/www/html/web/core/lib/Drupal/Core/DrupalKernel.php(1403): Symfony\Component\DependencyInjection\ContainerBuilder->get('http_kernel')
  #26 /var/www/html/web/core/lib/Drupal/Core/DrupalKernel.php(694): Drupal\Core\DrupalKernel->getHttpKernel()
  #27 /var/www/html/vendor/drush/drush/src/Boot/DrupalBoot8.php(326): Drupal\Core\DrupalKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Rende
  r\HtmlResponse))
  #28 [internal function]: Drush\Boot\DrupalBoot8->terminate()
  #29 {main}

  Next Symfony\Component\DependencyInjection\Exception\InvalidArgumentException: Cannot generate lazy proxy for service "http_middleware.big_pipe". in /var/www/html/vendor/symfony/
  dependency-injection/LazyProxy/PhpDumper/LazyServiceDumper.php:138
  Stack trace:
  #0 /var/www/html/vendor/symfony/dependency-injection/LazyProxy/Instantiator/LazyServiceInstantiator.php(33): Symfony\Component\DependencyInjection\LazyProxy\PhpDumper\LazyService
  Dumper->getProxyCode(Object(Symfony\Component\DependencyInjection\Definition), 'http_middleware...')
  #1 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1086): Symfony\Component\DependencyInjection\LazyProxy\Instantiator\LazyServiceInstantiator->instantiate
  Proxy(Object(Drupal\Core\DependencyInjection\ContainerBuilder), Object(Symfony\Component\DependencyInjection\Definition), 'http_middleware...', Object(Closure))
  #2 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\Dep
  endencyInjection\Definition), Array, true, 'http_middleware...')
  #3 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tru
  e)
  #4 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Componen
  t\DependencyInjection\Reference), Array, true)
  #5 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #6 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\Dep
  endencyInjection\Definition), Array, true, 'http_middleware...')
  #7 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tru
  e)
  #8 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Componen
  t\DependencyInjection\Reference), Array, true)
  #9 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #10 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\De
  pendencyInjection\Definition), Array, true, 'http_middleware...')
  #11 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tr
  ue)
  #12 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Compone
  nt\DependencyInjection\Reference), Array, true)
  #13 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #14 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\De
  pendencyInjection\Definition), Array, true, 'http_middleware...')
  #15 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tr
  ue)
  #16 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Compone
  nt\DependencyInjection\Reference), Array, true)
  #17 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #18 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\De
  pendencyInjection\Definition), Array, true, 'http_middleware...')
  #19 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1258): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_middleware...', 1, Array, tr
  ue)
  #20 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1210): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Object(Symfony\Compone
  nt\DependencyInjection\Reference), Array, true)
  #21 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(1110): Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices(Array, Array, true)
  #22 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(586): Symfony\Component\DependencyInjection\ContainerBuilder->createService(Object(Symfony\Component\De
  pendencyInjection\Definition), Array, true, 'http_kernel')
  #23 /var/www/html/vendor/symfony/dependency-injection/ContainerBuilder.php(531): Symfony\Component\DependencyInjection\ContainerBuilder->doGet('http_kernel', 1)
  #24 /var/www/html/web/core/lib/Drupal/Core/DrupalKernel.php(1403): Symfony\Component\DependencyInjection\ContainerBuilder->get('http_kernel')
  #25 /var/www/html/web/core/lib/Drupal/Core/DrupalKernel.php(694): Drupal\Core\DrupalKernel->getHttpKernel()
  #26 /var/www/html/vendor/drush/drush/src/Boot/DrupalBoot8.php(326): Drupal\Core\DrupalKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Rende
  r\HtmlResponse))
  #27 [internal function]: Drush\Boot\DrupalBoot8->terminate()
  #28 {main}
    thrown in /var/www/html/vendor/symfony/dependency-injection/LazyProxy/PhpDumper/LazyServiceDumper.php on line 138
catch’s picture

catch
he/him
Primary language English
commented

@godotislate maybe both so we can track it from each side?

Does it work if you remove the final from ContentLength?

godotislate’s picture

godotislate
he/him
commented

@godotislate maybe both so we can track it from each side?

Will do.

Does it work if you remove the final from ContentLength?

Yes, no exceptions thrown.

godotislate’s picture

godotislate
he/him
commented

Follow up: #3412160: Uncaught exception thrown when running database updates via drush
Drush issue: https://github.com/drush-ops/drush/issues/5848

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented

I've run into this before w/ drush on some other un-related operations. It assumes everything is non final and tries to wrap objects in its own wrapper via reflection. In the past we had to check for if the object was final. I wonder if that is what is happening here. The obvious thing is to not make the class final. But its awful convenient for a simple class like we have here to make it final and not need to worry about BC.

larowlan’s picture

larowlan
Location 🇦🇺🏝.au GMT+10
commented
Issue tags: -Patch release target

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

grimal’s picture

grimal commented

Hi,
I've encountere this issue after updating from version 10.1 to 10.2.3 on an OVH hosting platform (although everything it's fine in my local env).

While the page does render in the browser, it continues to load for approximately 5 seconds.

Upon testing the request with CURL, I observed that the HTML content is sent, but then there's a delay of 5 seconds before it returns the error: "curl: (18) transfer closed with 16932 bytes remaining to read". The CURL request executes without issue when the --compressed option is omitted.

To resolve this, I've removed 'Content-Length' header with $response->headers->remove('Content-Length') before calling $response->send() in the index.php file.

Any idea why I'm still experiencing this issue with Drupal 10.2.3, despite it should be fixed?

phily’s picture

phily
he/him
Primary language French
Location 🇪🇺🇫🇷 Paris, France
commented

@grimal: see #35 at issues 3419024 where the suggested addition to .htaccess file might help you.
Header always unset Content-Length