Turbo: Implement additive JavaScript aggregation to prevent multiple evaluation

Added more detail, links to relevant Drupal core and Turbo code, and an example of one hack that seems to work around the issue (though not fixing the underlying problem).

Setting to postponed for now; we can clearly state that a known limitation of this Turbo implementation is that JavaScript aggregation should be off or weird things will happen which should be fine for a dev or alpha release; when we start getting to beta, we'll need to implement the longer term proposed resolution in some way.

That may work, yeah, but I'm wondering if forcing a session for all users - including anonymous users - is going to have a noticeable performance hit. By default, Symfony doesn't start a session until there's something in the session to track (like in this case the Ajax page state) for performance reasons. That said, we do force a session for all users on Omnipedia to track the last few wiki pages a user has visited so that the random page controller knows not to pick something you've just seem, and that seems to perform well enough. If we do start sessions for all users, it would be a good idea to require the core page_cache module is uninstalled, because it'll prevent a session being started if a page is already cached - it took me far longer than I would have liked to figure out it was why our random page controller would sometimes redirect to the same page over and over for anonymous users.

I haven't messed around with any middleware as of yet so I'm not experienced with that but I like the idea. If anyone wants to create a basic prototype I'd be happy to test it out. Sorry about the slow replies; I'm busy with a bunch of stuff at the moment but checking in every so often.

So I tried to implement a solution based on @catch's idea (see the issue fork) and it sort of works but not quite. It doesn't yet have a cache context to vary by and it breaks Turbo since the Turbo JavaScript isn't found on the page so Turbo does a full page load, and I feel like piggybacking on ajaxPageState might not be worth the potential headaches down the road if that changes. That said, the overall idea seems good, but it needs a bunch of work to make it functional.

To do list/ideas

• Implement a cache context that varies based on a hash or compressed URL query string because we'll likely need to add that to the page #cache.
• Port over some of the 1.x refreshless_page_state logic and don't spoof or use ajaxPageState.
• Also don't forget RefreshlessRenderer::validatePreconditions() for security from the 1.x branch.
• Turbo and RefreshLess stuff needs to be handled separately and excluded from being removed on subsequent pages because doing so causes Turbo to relinquish control and let a full page load occur. We may have to decorate another one of the core asset services to group the RefreshLess and Turbo stuff into its own aggregate since setting it all to preprocess: false isn't great.
• Implement the PHP logic somewhere to exclude the already loaded libraries somewhere; our decorated AssetResolver would make the most sense since it's central to the whole process.

Core stuff to use as reference

\Drupal\big_pipe\Render\BigPipe::sendContent()

Example of dealing with cumulative asset libraries.

\Drupal\Core\Asset\AttachedAssetsInterface::setAlreadyLoadedLibraries()

Can be used to mark libraries as already loaded on a page and thus to not be loaded again.

\Drupal\Core\Asset\AssetResolver::getLibrariesToLoad()

Uses the above to remove already loaded libraries. We don't need to invoke this ourselves but just need to do the library marking as loaded before we hand off getCssAssets() and getJsAssets() to the decorated service.

\system_js_settings_alter()

Example of how ajaxPageState libraries are built.

\Drupal\Core\StackMiddleware\AjaxPageState::parseAjaxPageState()

Decompresses compressed libraries.

Alright, so I think I have it mostly working. It should now be aggregating and
attaching only what's not already on the page, with the following caveats/problems yet to be resolved:

• Our module JavaScript is set to preprocess: false for now because it doesn't seem possible to group ourselves into a separate aggregate; see #3232810: [PP-1] Allow setting aggregation groups for js files in library definitions. Ideally, we would be grouped into one aggregate that contains just the refreshless_turbo/refreshless and refreshless_turbo/turbo libraries so that we could take advantage of core's minification and gzipping.
• We have to special case the refreshless_turbo/refreshless and refreshless_turbo/turbo libraries to ensure Turbo is always attached even with the additive libraries logic, but never attached or aggregated more than once because bad things happen when Turbo loads and evaluates itself a second time.
• There's some duplicate logic between the middleware and our decorated asset resolver that I'm not happy with but I couldn't get it working well enough without it. I'll probably rewrite some of it into an attachments processor (see the HTMX module's HtmxResponseAttachmentsProcessor) to hopefully remove the duplication.
• This re-introduces the same CSS specificity symptoms as seen in #3399314: Turbo: CSS files merged into the <head> always appended instead of preserving full page load order, causing CSS specificity issues but now when aggregation is enabled as well: the cause now seems to be that the base theme's (Claro) CSS for dropbuttons and vertical tabs is additively aggregated (since they're defined in separate libraries and not on every page) while Gin compiles dropbuttons into its big gin.css that gets attached to every page and uses a fixed weight value on its tabs.css library definition rather than listing Claro's as a dependency. The best course of action may be to rework the additive aggregation so that it only applies to JavaScript and not libraries as a whole, leaving CSS to aggregate as it did before, non-additively; this may be doable via our decorated asset resolver in the getCssAssets() method by checking if the current request is a Turbo request and then temporarily removing the already loaded libraries, calling $this->decorated->getCssAssets(), then restoring the already loaded libraries before returning the CSS assets.
• It's unclear if this needs a cache context because this seems to be working with caching, but my gut is saying that we should have a cache context and set up a clear distinction between a full page load and a Turbo/additive load, because there's a bunch of stuff in the code that just YOLOs it without a care which might backfire in the future.

So it still needs work but some progress has been made. The hardest part of this is working without an easy way to visualize what libraries are being (additively) aggregated and attached to a page. Setting up some tests for that would help, but I would also want to implement console logging of what new libraries get added, much like the 8.x-1.x branch has, and may something a bit more visual would help as well.

Updating title as we now know what needed to be done and why.

Heavy rewrite of issue summary reflecting current understanding of the problem and solution.

Added remaining tasks and follow up tasks.

Minor grammar fix.

Updated status of a few items.

Reordered "Remaining tasks" and "Follow up tasks", and moved some stuff between them where they make more sense.

Link to #3401146: Turbo: Automated tests for test details.

Added a few remaining tasks from my comments.

Added References section with links to the HTMX module.

Removed session item (not actually true) and added page cache request policy item.

I've created a new branch (3414538-what-if-attachments-processor) in the issue fork to try and implement a decorated attachments processor, but I can't yet get it to work. The main motivation here is to no longer rely on ajaxPageState because outputting it on every page results in certain generated URLs that have a query string (e.g. pager, admin/config compact toggle link, etc.) also including the Ajax page state in their URL, which in turn causes some weird behaviour with those links where Turbo doesn't kick in during history back/forward navigation, in addition to looking messy and ugly.

There's something I'm not understanding and not doing correctly with the library diffing, and without a clear indicator of what libraries are being added and when, it's hard to pin down. I think this would need some kind of debugging output when libraries are added to hopefully spot what's going wrong, but I'm tired and need to do other stuff right now.