Using Sessionless BigPipe to ALWAYS serve using BigPipe and NEVER from PageCache when using BigPipe Paragraphs (or other functionality)

Interesting, I've never heard of https://www.drupal.org/project/big_pipe_paragraphs until now! 😃

I find that the placeholder appears on the first load after clearing cache, but on subsequent loads the paragraph is already fully rendered.

This is when testing as the anonymous user, right?

I do see Big pipe paragraphs working correctly for authenticated users.

Ok, then the answer must be "yes" to my previous question. 👍

I'm afraid you're misunderstanding the goal of https://www.drupal.org/project/big_pipe_sessionless and perhaps even of BigPipe. 😅

The goal is not to animate things or to make things appear sequentially.

The goal is to make the content that the end user wants to see appear as fast as possible.

BigPipe is never active for sessionless visits (typically these are anonymous visitors) by default. This module (Sessionless BigPipe) makes it possible to use it for those visits nonetheless, hence making those visits see content faster. But since all sessionless visits get the exact same content, there is literally no point in doing this over and over again. Drupal ships with "Page Cache", which makes page loads instantaneous for sessionless visits (anonymous users). Well … Sessionless BigPipe therefore chooses to populate the Page Cache after that first anonymous visit! This means you get the best of both worlds.

In other words: what you're describing/perceiving as a problem is absolutely the intended behavior, and is what makes for the fastest user experience.

So … I'm wondering why you think this is wrong/a problem? 🤔 😊

What I think he is looking for is to load certain content after the initial page load because it is dynamic. It is not depending than on the session of the user. Another thing is perhaps (that was also the original idea of big pipe paragraphs, yes I made it) load large uncached (but cacheable) pages faster by defer loading of paragraphs or blocks below the fold to a later moment. To increase the speed to serve these pages.

For instance, I have a page with random teasers of articles that I want to show. Everything on the page is static and can be cached for the anonymous user. Except this block with random teasers. But because it's random you don't/won't/shouldn't cache the block because well, it needs to randomize. Making also the rest of the page uncacheable.

How can we show these uncacheable blocks through placeholders on pages for anonymous users?

By default this does complicate using CDN's/varnish like solutions that will not send requests to the webserver if it already has the page cached and the current user has no session. It will only go through to the webserver if the user has some dynamic signature, a session. But that does sound somewhat limited right? Why can't we use similar ajax replacement as Big pipe uses to send out a request to replace those blocks after the page conent is loaded?
As I see it you should have some kind of solution like these in drupal 7:
https://www.drupal.org/project/ajaxblocks
https://www.drupal.org/project/authcache (block submodule)

It might not be ideal, because it won't be as effective as big pipe that pushes the ajax responses through in the same response stream. But it can still make an uncacheable page perhaps cacheable.

This issues seems somewhat related. https://www.drupal.org/project/drupal/issues/2828136

To perhaps even clarify further. It's for similar reasons you want to lazy load images that are not shown above the fold. It's just not a static asset, but some dynamic content to be retrieved from Drupal.

load large uncached (but cacheable) pages faster by defer loading of paragraphs or blocks below the fold to a later moment

Everything on the page is static and can be cached for the anonymous user. Except this block with random teasers.

Blocks are already placeholderable, and hence BigPipe can and does already deliver these later.

What is an example of a paragraph that merits being loaded later? Are there paragraphs that are dynamically computed?

How can we show these uncacheable blocks through placeholders on pages for anonymous users?

Please define "uncacheable". Is it session-specific? Then just using BigPipe itself will already work.

It will only go through to the webserver if the user has some dynamic signature, a session. But that does sound somewhat limited right? Why can't we use similar ajax replacement as Big pipe uses to send out a request to replace those blocks after the page conent is loaded?

Right, it is somewhat limited, but intentionally so: to balance performance for the individual visitor against the increased load on the server. Using AJAX requests can work, but it means that instead of having to serve 1 request per visited page (1:1 relationship), the web server must serve 1 + x AJAX requests (1:N relationship).

That's why Drupal core will never do the approach you suggest out of the box: it risks making many sites exceed their hosting's technical or budgetary limitations. BigPipe as it is implemented right now does not impose extra infrastructure costs, and that's why it's so appealing: an improvement in many scenarios (but indeed not all!), without a corresponding cost increase 😊

Blocks are already placeholderable, and hence BigPipe can and does already deliver these later.

But only if there is a session running.

Please define "uncacheable". Is it session-specific? Then just using BigPipe itself will already work.

Like random articles to show for some reason.

Right, it is somewhat limited, but intentionally so: to balance performance for the individual visitor against the increased load on the server.

From a "smaller" site standpoint I totally agree with this, and think that this is how it should work.

But for larger platforms you run into a lot cache, that will be cleared out to make space for other cache items. So you want more granular control on blocks/components in your page that you want to cache and stuff that you could load in with a bit of a delay and you can keep uncached. Improving the perception of the speed on your site for the end user.

Like the footer, you get a lot of render cache for this because the menu trail can often change. Increase the use of the cache space in your DB or other cache storage. But this footer is often invisible for users and only shown when scrolling down. How awesome would it be if you could set a block to load later and not cache it. Still making the site seem fast and keep other data still in cache and not being invalidated because of it.

The js ajax solution I propose is ofcourse a bit of a quick and dirty solution that you might not want. But another one is a varnish version with a somewhat similar tactic, only varnish does the replacing:
https://github.com/ThijsFeryn/drupal8_esi_placeholders
Don't know if this works though.

But making a module that also supplies the JS version of my proposal is something that should fit in a different contrib module I think.

But only if there is a session running.

Correct. Because when there is no session, Drupal assumes it's an anonymous user.

Like random articles to show for some reason.

Okay, so "random on every request", right?

Increase the use of the cache space in your DB or other cache storage. But this footer is often invisible for users and only shown when scrolling down. How awesome would it be if you could set a block to load later and not cache it.

Oh, yes, this I completely agree with! A flag to signal "this is not worth caching", right? Or better yet: automatically only cache things that cost X CPU cycles and/or Y DB queries.

But this module is not the place for that. The cache system in Drupal core is.

There is an issue for this already, but I can't quickly find it. I'm sure it's linked somewhere from #2429287: [meta] Finalize the cache contexts API & DX/usage, enable a leap forward in performance. #2498857: Support min-age in render caching is somewhat related.

The js ajax solution I propose is ofcourse a bit of a quick and dirty solution that you might not want.

It's not just quick and dirty. It makes things slower in aggregate and increases server load.

But another one is a varnish version with a somewhat similar tactic, only varnish does the replacing:
https://github.com/ThijsFeryn/drupal8_esi_placeholders

It can work. But like you say: it requires extra infrastructure. There is a core issue to bring this to Drupal core: #2524452: [meta] The ESI battle plan [v2].

But making a module that also supplies the JS version of my proposal is something that should fit in a different contrib module I think.

Oh, absolutely!

I think it'd be great if there were an ajax_placeholders module, which introduces a new \Drupal\Core\Render\Placeholder\PlaceholderStrategyInterface implementation, which uses AJAX instead! And there's no reason why this cannot co-exist with BigPipe and Sessionless BigPipe: they can all be used together!

Re-read this today.

I see where you're coming from. I understand that in very specific scenarios, what you're asking for would help. But in most cases, it would not.

I find that the placeholder appears on the first load after clearing cache, but on subsequent loads the paragraph is already fully rendered.

👆 This means that this module works fine already. That's literally the behavior described on the project page 😄

What you're asking for, is a different behavior. You're asking to not prime the page cache even when it could be primed.

And in fact … this is already supported 🤓

1. Eventually, Sessionless BigPipe calls \Drupal\big_pipe_sessionless\Render\BigPipeSessionless::primePageCache() with a HtmlResponse object to prime the Page Cache with.
2. That calls \Drupal\page_cache\StackMiddleware\PageCache::storeResponse(), which contains this logic:
   

       // Allow policy rules to further restrict which responses to cache.
       if ($this->responsePolicy->check($response, $request) === ResponsePolicyInterface::DENY) {
         return FALSE;
       }
   

3. In other words: the Page Cache will not be primed if BigPipe Paragraphs were to define a response policy that detects whether the response contains rendered BigPipe Paragraphs placeholders … which would give you exactly the behavior you're asking for 😊

Ensured #8 would not go to waste by opening #3396097: Allow configuring to *never* store in Page Cache when using Sessionless BigPipe. 👍