Optimize ajaxPageState to keep Drupal 8 sites fast on high-latency networks, prevent CSS/JS aggregation from taking down sites and use HTTP GET for AJAX requests

Two child issues are now reviewable already:

1. #2377397: Themes should use libraries, not individual stylesheets
2. #2378095: Convert all remaining attached individual CSS/JS assets to attached asset libraries

Another child issue created that is reviewable already: #2378263: hook_library_alter() must be manually invoked by users of LibraryDiscovery, and has no test coverage.

Note that the two child issues mentioned in #8 are now green and effectively blocked on review.

Another child issue: #2378789: Views output cache is broken. The first three are all green.

Now tracking child issues in the IS.

3 of the 4 child issues have landed. Time to open some more :)

Opened:

1. (not blocked, patch upcoming) #2382557: Change JS settings into a separate asset type
2. (blocked on the aforementioned issue and the other child issue that is not yet committed, #2378095) #2382533: Attach assets only via the asset library system

Oh, thank you!

#2378095: Convert all remaining attached individual CSS/JS assets to attached asset libraries landed.

Now only blocked on #2382557: Change JS settings into a separate asset type, where consensus was reached but I just need to refactor it based on the consensus, and on #2382533: Attach assets only via the asset library system, which itself is blocked on that one too.

Almost there! :)

#2382557: Change JS settings into a separate asset type was committed!

All true.

Two buts:

• surprisingly many sites don't send HTML gzipped.
• several KB of poorly compressible data is still going to cost you.

Of course this isn't a silver bullet. But it should help, and if we don't do it now, we won't be able to do it later. Thanks for helping keep ourselves honest though :)

#2378095: Convert all remaining attached individual CSS/JS assets to attached asset libraries just landed, so now this issue is fully unblocked! :) Working on this patch now…

#26: done! Also tried to improve the title.

Finally, a patch. Diffstat:

39 files changed, 1371 insertions(+), 1204 deletions(-)

Updated the IS with the API changes.

(Note: the IS currently says Drupal 8 sends 4.3 KB of AJAX page state; that's now gone up to 4.9 KB…

Before

Frontpage, as user 1: 4876 bytes:
"js":{"core\/assets\/vendor\/html5shiv\/html5.js":1,"core\/assets\/vendor\/jquery\/jquery.min.js":1,"core\/assets\/vendor\/domready\/ready.min.js":1,"core\/misc\/drupal.js":1,"core\/assets\/vendor\/underscore\/underscore-min.js":1,"core\/assets\/vendor\/backbone\/backbone-min.js":1,"core\/assets\/vendor\/modernizr\/modernizr.min.js":1,"core\/assets\/vendor\/jquery-once\/jquery.once.js":1,"core\/modules\/contextual\/js\/contextual.js":1,"core\/modules\/contextual\/js\/models\/StateModel.js":1,"core\/modules\/contextual\/js\/views\/AuralView.js":1,"core\/modules\/contextual\/js\/views\/KeyboardView.js":1,"core\/modules\/contextual\/js\/views\/RegionView.js":1,"core\/modules\/contextual\/js\/views\/VisualView.js":1,"core\/assets\/vendor\/jquery-form\/jquery.form.js":1,"core\/assets\/vendor\/jquery.ui\/ui\/jquery.ui.position.js":1,"core\/misc\/debounce.js":1,"core\/misc\/displace.js":1,"core\/assets\/vendor\/jquery.cookie\/jquery.cookie.min.js":1,"core\/misc\/form.js":1,"core\/misc\/progress.js":1,"core\/misc\/ajax.js":1,"core\/assets\/vendor\/jquery.ui\/ui\/jquery.ui.core.js":1,"core\/assets\/vendor\/jquery.ui\/ui\/jquery.ui.widget.js":1,"core\/assets\/vendor\/jquery.ui\/ui\/jquery.ui.button.js":1,"core\/assets\/vendor\/jquery.ui\/ui\/jquery.ui.mouse.js":1,"core\/assets\/vendor\/jquery.ui\/ui\/jquery.ui.draggable.js":1,"core\/assets\/vendor\/jquery.ui\/ui\/jquery.ui.resizable.js":1,"core\/assets\/vendor\/jquery.ui\/ui\/jquery.ui.dialog.js":1,"core\/misc\/dialog\/dialog.js":1,"core\/misc\/dialog\/dialog.position.js":1,"core\/misc\/dialog\/dialog.jquery-ui.js":1,"core\/modules\/quickedit\/js\/quickedit.js":1,"core\/modules\/quickedit\/js\/util.js":1,"core\/modules\/quickedit\/js\/models\/BaseModel.js":1,"core\/modules\/quickedit\/js\/models\/AppModel.js":1,"core\/modules\/quickedit\/js\/models\/EntityModel.js":1,"core\/modules\/quickedit\/js\/models\/FieldModel.js":1,"core\/modules\/quickedit\/js\/models\/EditorModel.js":1,"core\/modules\/quickedit\/js\/views\/AppView.js":1,"core\/modules\/quickedit\/js\/views\/FieldDecorationView.js":1,"core\/modules\/quickedit\/js\/views\/EntityDecorationView.js":1,"core\/modules\/quickedit\/js\/views\/EntityToolbarView.js":1,"core\/modules\/quickedit\/js\/views\/ContextualLinkView.js":1,"core\/modules\/quickedit\/js\/views\/FieldToolbarView.js":1,"core\/modules\/quickedit\/js\/views\/EditorView.js":1,"core\/modules\/quickedit\/js\/theme.js":1,"core\/assets\/vendor\/classList\/classList.min.js":1,"core\/misc\/active-link.js":1,"core\/modules\/views\/js\/views-contextual.js":1,"core\/misc\/announce.js":1,"core\/assets\/vendor\/matchMedia\/matchMedia.min.js":1,"core\/assets\/vendor\/matchMedia\/matchMedia.addListener.min.js":1,"core\/modules\/toolbar\/js\/toolbar.menu.js":1,"core\/modules\/toolbar\/js\/toolbar.js":1,"core\/modules\/toolbar\/js\/models\/MenuModel.js":1,"core\/modules\/toolbar\/js\/models\/ToolbarModel.js":1,"core\/modules\/toolbar\/js\/views\/BodyVisualView.js":1,"core\/modules\/toolbar\/js\/views\/MenuVisualView.js":1,"core\/modules\/toolbar\/js\/views\/ToolbarAuralView.js":1,"core\/modules\/toolbar\/js\/views\/ToolbarVisualView.js":1,"core\/assets\/vendor\/jquery-joyride\/jquery.joyride-2.0.3.js":1,"core\/modules\/tour\/js\/tour.js":1,"core\/misc\/tabbingmanager.js":1,"core\/modules\/contextual\/js\/contextual.toolbar.js":1,"core\/modules\/contextual\/js\/toolbar\/models\/StateModel.js":1,"core\/modules\/contextual\/js\/toolbar\/views\/AuralView.js":1,"core\/modules\/contextual\/js\/toolbar\/views\/VisualView.js":1,"core\/modules\/toolbar\/js\/escapeAdmin.js":1},"css":{"contextual.module.css":1,"contextual.theme.css":1,"contextual.icons.css":1,"jquery.ui.core.css":1,"jquery.ui.theme.css":1,"jquery.ui.button.css":1,"jquery.ui.resizable.css":1,"jquery.ui.dialog.css":1,"dialog.theme.css":1,"quickedit.module.css":1,"quickedit.theme.css":1,"quickedit.icons.css":1,"quickedit.css":1,"normalize.css":1,"system.module.css":1,"system.theme.css":1,"layout.css":1,"elements.css":1,"typography.css":1,"admin.css":1,"book.css":1,"breadcrumb.css":1,"captions.css":1,"comments.css":1,"content.css":1,"contextual.css":1,"dropbutton.component.css":1,"featured.css":1,"footer.css":1,"form.css":1,"forum.css":1,"header.css":1,"help.css":1,"highlighted.css":1,"list.css":1,"messages.css":1,"node-preview.css":1,"pager.css":1,"primary-menu.css":1,"search.css":1,"search-results.css":1,"secondary-menu.css":1,"shortcut.css":1,"skip-link.css":1,"sidebar.css":1,"table.css":1,"tabs.css":1,"tips.css":1,"triptych.css":1,"user.css":1,"vertical-tabs.component.css":1,"views.css":1,"buttons.css":1,"media.css":1,"colors.css":1,"print.css":1,"views.module.css":1,"toolbar.menu.css":1,"toolbar.module.css":1,"toolbar.theme.css":1,"toolbar.icons.css":1,"user.icons.css":1,"tour.module.css":1,"contextual.toolbar.css":1,"shortcut.module.css":1,"shortcut.theme.css":1,"shortcut.icons.css":1}

After

Frontpage, as user 1: 356 bytes:
"libraries":"bartik\/global-styling,classy\/base,contextual\/drupal.contextual-links,contextual\/drupal.contextual-toolbar,core\/drupal.active-link,core\/html5shiv,quickedit\/quickedit,shortcut\/drupal.shortcut,system\/base,toolbar\/toolbar,toolbar\/toolbar.escapeAdmin,tour\/tour,user\/drupal.user.icons,views\/views.contextual-links,views\/views.module"

That's 4.5 KB less data to send on a stock Standard profile install on the front page. The new asset AJAX page state data is 356/4876=7.3% of the original, a 92.7% improvement. The total AJAX page state (including theme metadata) is then going to be 4971-4876+356=451 bytes, so 451/4971=9.1% of the original.
And this is on stock Drupal; the difference is going to be even starker on real sites!

This also happens to be a great way to spot problems: libraries that are unexpectedly listed there either:

• should not be loaded at all (e.g. views/views.module, only contains table and grid style styling, neither of which is used for the frontpage view) and hence expose front-end performance problems
• should be a dependency of another library (e.g. classy/base is implicitly a dependency of bartik/global-styling)

I.e. a net DX improvement.

For those of you wondering why I'm not listing every library: that's done for a good reason. See the "minimal representative subset" code, also explained in the IS:

The minimal representative subset is what is crucial in sending as little information as possible to the client-side: e.g. if you attach the core/drupal.autocomplete, core/jquery and core/jquery.ui.position libraries, then we're going to load those two libraries plus all of their dependencies. But the dependencies of core/drupal.autocomplete are core/jquery, core/drupal, core/drupal.ajax, core/jquery.ui.autocomplete, and the dependencies of core/jquery.ui.autocomplete are core/jquery.ui, core/jquery.ui.widget, core/jquery.ui.position and core/jquery.ui.menu. We don't want to list all of those dependencies: that'd be both space consuming and unnecessary: we can just list core/drupal.autocomplete and send that to the server with each AJAX request, the server (Drupal) then knows that all dependencies have been loaded, it doesn't need us to send the full list. Also, the second library that we attached was core/jquery, but that's actually already a dependency of core/drupal.autocomplete, so we don't need to list that either. And finally, core/jquery.ui.position is *also* a dependency of core/drupal.autocomplete, although an indirect one. Nevertheless, that means we don't need to list that either.
Therefore the minimal representative subset in this example is core/drupal.autocomplete. That's right: that single library is sufficient! In computer science/graph theory terms: each asset library has dependencies and can therefore be seen as a tree. Hence the given list of libraries represent a forest. This function returns all roots of trees that are not a subtree of another tree in the forest.

This doesn't address #32 yet.

A few of the failures were oversights on my part. But most of them were due to an API change that I hadn't yet mentioned in the IS: as of this patch, ajaxPageState is only added if it's actually necessary, i.e. if the core/drupal.ajax library is loaded. This makes AJAX page state compliant with the overarching rule: we only attach drupalSettings if an attached library depends on it.

Most of the test failures in #29 occurred because they relied on ajaxPageState being present always, and absent in certain test case scenarios in DialogTest, ViewAjaxTest and RowUITest that are very synthetic. They were extremely painful to fix.

A few of the test failures were due to areas I'd overlooked: the OpenDialogCommand still attached the core/drupal.dialog.ajax library via drupal_process_attached(), to put it in the _drupal_add_js() static, but that of course doesn't work anymore.

We should really untangle the mess that is the ajax_test test module, because the incredible hacks upon hacks that we've gathered there over time make it an absolute hell to fix any test failures in that area. Not doing that here, because the patch is already big enough.

A few problems that were spotted in the current patch, fixed in this reroll:

1. +++ b/core/core.services.yml
   @@ -1074,6 +1074,13 @@ services:
      library.discovery.parser:
        class: Drupal\Core\Asset\LibraryDiscoveryParser
        arguments: ['@app.root', '@module_handler']
   +    public: false
   

   Problem introduced during rebasing. Will remove that change.

2. +++ b/core/lib/Drupal/Core/Asset/AssetResolver.php
   @@ -0,0 +1,475 @@
   +            case 'inline':
   ...
   +            case 'inline':
   

   These switch statement cases need to be removed; they only exist because of a faulty rebase from long ago.

1. I know. But the impact is extremely low, and it makes a lot of long-term sense: if we want to keep drupal_get_js(), then we have to keep the kludgy+fragile system we've been trying to get rid of around to some degree just for the very few people — if any — that are already using it in 8. This is a <1% API function.
2. Done.
3. First: I agree that it's sad this lives in a preprocess function. But I think the entire template_preprocess_html() function is sad; we should move that logic into a HtmlResponse class, or at least let HtmlRenderer::renderResponse() take care of this. But doing *that* is out of scope here.
   Second: these checks used to be in the #pre_render callbacks of \Drupal\Core\Render\Element\(Styles|Scripts). But #type => '(styles|scripts)' no longer makes sense to have, since A) all it did was return the output of a service, B) it was only used inside drupal_get_(css|js)().
   Did that answer it?
4. Excellent question! I've also considered that approach, but I just wanted to get something that works done ASAP. This is exactly the sort of thing I thought/hoped would be criticized.
   I think the only thing preventing us from going with a stateless service and then sending the service the necessary state, is the advanced use case in system_js_settings_alter():

     // Now generate the values for the core/drupal.ajax library.
     // We need to send ajaxPageState settings for core/drupal.ajax if:
     // - ajaxPageState is being loaded in this Response, in which case it will
     //   already exist at $settings['ajaxPageState'] (because the core/drupal.ajax
     //   library definition specifies a placeholder 'ajaxPageState' setting).
     // - core/drupal.ajax already has been loaded and hence this is an AJAX
     //   Response in which we must send the list of extra asset libraries that are
     //   being added in this AJAX Response.
     /** @var \Drupal\Core\Asset\AssetResolverInterface $asset_resolver */
     $asset_resolver = \Drupal::service('asset.resolver');
     if (isset($settings['ajaxPageState']) || in_array('core/drupal.ajax', $asset_resolver->getAlreadyLoadedLibrariesWithDependencies())) {
       // Provide the page with information about the theme that's used, so that
       // a later AJAX request can be rendered using the same theme.
       // @see \Drupal\Core\Theme\AjaxBasePageNegotiator
       $theme_key = \Drupal::theme()->getActiveTheme()->getName();
       $settings['ajaxPageState']['theme'] = $theme_key;
       // Checks that the DB is available before filling theme_token.
       if (!defined('MAINTENANCE_MODE')) {
         $settings['ajaxPageState']['theme_token'] = \Drupal::csrfToken()->get($theme_key);
       }
   
       // Provide the page with information about the individual asset libraries
       // used, information not otherwise available when aggregation is enabled.
       $library_dependency_resolver = \Drupal::service('library.dependency_resolver');
       $minimal_libraries = $library_dependency_resolver->getMinimalRepresentativeSubset(array_unique(array_merge(
         $asset_resolver->getLibrariesWithDependencies(),
         $asset_resolver->getAlreadyLoadedLibrariesWithDependencies()
       )));
       sort($minimal_libraries);
       $settings['ajaxPageState']['libraries'] = implode(',', $minimal_libraries);
   

   There are two reasons for it to live there:

   1. Pattern: It's analogous to how we expand the core/drupalSettings library's drupalSettings, to replace it with the necessary dynamic values.
   2. Independence: to generate the AJAX page state, we MUST know what asset libraries are being loaded for this response, and which the client already has. But generating the AJAX page state from within AssetResolver itself — which is the only possible alternative if we don't want to expose the state of AssetResolver — is at least equally ugly, because it's the content of a specific asset, and the AssetResolver should only deal with the libraries-to-concrete-assets logic, not also generate the content of a specific asset.

   Yes, this means that we still have a global state. But only for the very briefest of moments: when #type => html is being rendered, until the asset alter hooks have been fired.

   That being said, I think the ideal would indeed be to make AssetResolver stateless. I think the ideal would look like this: add a ResponseWithAssets class that subclasses Symfony's Response, which must be subclassed by any response that wants to attach assets (HtmlResponse, AjaxResponse …). It'd allow libraries and settings to be set on the response object. We could then have a KernelEvents::RESPONSE event subscriber with a low priority, that'd run before the response is actually rendered, to map asset libraries to concrete assets. It'd call the stateless AssetResolver, which would accept ResponseWithAssets objects as input. And finally, with only a single API addition — also providing the Response object to hook_(css|js|js_settings)_alter(), AJAX page state could just use the data on there, call AssetResolver with it there and … voila!
   Thoughts? (It's late here so I'm probably missing something.)

5. Can you explain that? AFAICT Turbolinks would use AJAX page state also, or something like it. If you're on page X which contains libraries A and B, and you're going to page Y using Turbolinks, and that should contain libraries B and C, then you're going to want to only load library C.
   (Unless you're talking about "unloading" libraries… which shouldn't be necessary thanks to our Drupal.behaviors abstraction.)
   EDIT: oh, wait, I see what you mean. You're saying that in case of Turbolinks, the set of already loaded libraries would not be the empty set. Easy fix: check if ajax_page_state is available, and if so, use it. Fixed.
6. I think you meant "take" instead of "make"? :)
   If so, then I think: no. I think it's better to be explicit here. Render arrays are a vaguely defined, almost freeform data structure. AssetResolver(Interface) and LibraryDependencyResolver(Interface) work with well-defined data structures. I think it's better to provide exact input, to not pollute those interfaces/classes with vague methods just to simplify a handful of calls.
7. You have to look at the bigger context of the 3 LoC you quoted. Basically, AjaxResponse was doing VERY VERY … unethical things in order to have _drupal_add_(css|js)() contain only those assets that hadn't been already loaded. Basically, AjaxResponse implemented part of the logic of drupal_get_(css|js)() to modify their internal state beforehand, and then it called them with the $skip_alter parameter set to TRUE. If that's not mind-bending, then what is? :P And now it's GONE!
8. LOL, Wim--. Fixed.
9. The idea is that AssetResolver is only called once per request. So unless I'm misunderstanding you, I don't think that makes sense.
10. Not currently, but because it's theoretically possible to call AssetResolver multiple times, e.g. in asset library-analyzing code, I added it. If you want to remove it, I could live with that, but it seems prudent to be able to reset state.
11. Good catch; fixed.
12. :)
13. Done.

#35:

So I wonder whether we could pass along the object in the alter hook. It is certainly part of that domain already.

That's a good point. That actually means we can separate state already; the outline of "the ideal" I provided is then not a blocker. Thanks! :)

That kind of logic is really something you would eventually like to override.

Agreed, but that'd require that "ideal".

[…] note the idea of passing around the storage would solve that at the same time.

Agreed!

Trying this right now…

#36: Thanks!

Implemented #35.

What do you think, dawehner? I think I like it better, but I'm not fond of the name: AttachedAssets(Interface) seems like it could be better named.

And once we have that, I think this is another logical incremental improvement.

Removing debug leftovers…

PHP 5.4 sigh.

Straight reroll of #46. #2303761: Move views_ajax_form_wrapper() to ViewsFormBase broke this. Next: addressing the feedback.

#47: :)

#48: :)

#49:

1. I answered this already in #34.3. I'm not sure what you want me to do?
2. Done.
3. First Q: We could definitely do that. But this patch is already big. Can't we do that in a follow-up?
   Second Q: good question. I kept the same logic as _drupal_add_library() had. See more below, as part of the next point.
4. getLibrariesWithDependencies() returns a list, not a set! From the docs:
   

      * @return string[]
      *   A list of libraries, in the order they should be loaded, including their
      *   dependencies.
   

   This is used for determining which libraries to actually load, and loading libraries means loading assets, and that's sensitive to order, hence: list, not set.
   When determining which things to load, we must therefore always use lists (since order matters), but when determining which things are already loaded, sets suffice (since order doesn't matter; we only need to know if a given library was loaded or not). Hence getMinimalRepresentativeSubset()'s docs say this:

      * @return string[]
      *   A representative subset of the given set of libraries.
   

5. Thanks for the link! For others wanting to read it: most of that paper focuses on the weighted set-cover problem, but this only needs a good algorithm for the basic set-cover problem. You can skip anything related to the weighted set-cover problem; or just assume the weight is the same for all sets.
   First: the set-cover problem is NP-complete: https://en.wikipedia.org/wiki/Set_cover_problem.
   Second: that same Wikipedia page also mentions the greedy algorithm as being the best choice.
   Third: I'd argue that we can improve the algorithm performance later, if it turns out to be a problem. I think we should go with the easiest to understand code.

   The current algorithm looks like this:

   LET minimal = EmptySet()
   
   # Iterate over all libraries
   FOREACH lib IN libraries
   
     # Check if the current library is a dependency of another library
     LET exists = false
     FOREACH (other_lib, dependencies) IN other_libraries
       IF lib IN dependencies
         exists = TRUE
         BREAK
   
     # If it's not, then add it to the minimal set.
     IF NOT exists
       minimal[] = lib
   

   The greedy algorithm (at each stage, choose the set that contains the largest number of uncovered elements) would look like this:

   LET minimal = EmptySet()
   LET is_covered = false
   LET covered_dependencies = EmptySet()
   LET all_dependencies_to_cover = EmptySet()
   
   # First determine what the set of dependencies to cover is; i.e. what universe to strive for.
   FOREACH (lib, dependencies) IN libraries
     all_dependencies_to_cover =  all_dependencies_to_cover UNION dependencies
   
   # Now, find the minimal set cover.
   WHILE NOT covered_dependencies == all_dependencies_to_cover
     # Step 1: Calculate how many uncovered dependencies each library would cover.
     LET uncovered_deps_per_lib = EmptyHash()
     FOREACH (lib, dependencies) IN libraries
       uncovered_deps_per_lib[lib] = |dependencies - covered_dependencies|
   
     # Step 2: Choose the library in uncovered_deps_per_lib with the highest value.
     LET max = 0
     LET next_lib = null
     FOREACH lib IN libraries
       IF uncovered_deps_per_lib[lib] > max
         max = uncovered_deps_per_lib[lib]
         next_lib = lib
   
     # Step 3: Add the chosen library to covered_dependencies, and go to the next iteration
     minimal[] = next_lib
     covered_dependencies = covered_dependencies UNION dependencies
   

   Not only does it look more complex, it's also more difficult to grok, because it's no longer directly using properties of the concepts we're dealing with: asset libraries. Asset libraries are fundamentally trees, not sets. By looking at it from a trees angle, as the current algorithm does (and as the documentation indicates), it keeps the algorithm simpler, IMO.

#50: Great remark! I'll be very honest: that's what I originally wanted to do. But that means AjaxResponse must use drupal_merge_attached(). And since that is still a procedural function, it has a big ripple effect: many AJAX unit tests will have to start mocking that. Since this patch already is very big, I wanted to leave that for that issue.
Furthermore, and possibly an even stronger argument: I have not ever seen a single AJAX response with multiple HTML responses. So the actual value of that would be rather low. And in those rare cases, the logic building the AJAX response could just as well do the necessary merging, by calling drupal_merge_attached() itself? :)
In any case, I'd like to get this in and then consider adding that.

The weird whitespace in #52 is completely the fault of d.o's filter setup being broken, see #2396165: Code filter configuration removes blank lines.

#54: I could file issues for looking into algorithmic optimizations and getting rid of theme_preprocess_html(). Did you want to see other issues filed too, or?

#55: yep, sure, that's possible :) The storage mechanism doesn't matter for this issue; what matters is which actual data needs to be sent.

Created the requested follow-ups:

1. #2407187: Optimize LibraryDependencyResolver::getMinimalRepresentativeSubset() and win >=4%
2. #2407195: Move attachment processing to services and per-type response subclasses
3. #2407201: Consider introducing AjaxResponse::addAttachments()

Chasing HEAD, #2403269: Update to jQuery UI 1.11.2 broke this.

Note the diffstat: 58 files changed, 1494 insertions(+), 1299 deletions(-) — 711 lines are removed from common.inc, bringing that down from 2709 LOC to 1998.

(All the while bringing full unit test coverage for resolving library dependencies, which has zero explicit test coverage in HEAD.)

So nod_ says he can't sign off on the PHP. dawehner says the same.

Pinged Moshe for a review.

In the mean time, all I can do to push this issue forward, is to review it myself for nitpicks and fix those. And I found a whole bunch :) Now the diffstat has gotten even better: 58 files changed, 1471 insertions(+), 1299 deletions(-).

Updated the IS. Next: creatinga CR.

CR created; minor changes in IS, to improve clarity.

Straight reroll of the RTBC'd #64, and hence putting back at RTBC.

#73/#75: AFAICT what you're describing is impossible. I think you're testing with an improperly patched site.
Either the ajax_page_state value is present in the HTTP POST data, or it's not. If it's present, then it contains a 'libraries' key. Your problem description says that the ajax_page_state value is present, but it doesn't contain a 'libraries' key. This is impossible.
Therefore the existing check is sufficient. If it turns out I'm wrong, then we can very easily fix it in a follow-up issue.

1. Yes, clarified. Also fixed the doc example just below it.
2. You're right, that @return doc was confusing, clarified.
3. This is copied verbatim from drupal_get_css(): since I kept the same logic, I also kept the comments. But, per your request, updated it.
4. Fixed.

Also made one tiny change in theme.inc, at chx' request: retrieve a service once, assign it to a variable, instead of retrieving the same service twice.

Since this reroll only changed comments for all intents and purposes, leaving at RTBC.

#80: np, thanks :) All addressed in this reroll! Still keeping at RTBC, because again only comment changes.

I cannot reproduce the failure in #79 locally, assuming it was a random fail for now.

YAY! Thanks for the commit! Next: those requested follow-ups, all of which will be non-critical :)

#36: actually, JSON_UNESCAPED_SLASHES is what we want :) But, turns out this option is not turned on for a good reason: http://stackoverflow.com/questions/1580647/json-why-are-forward-slashes-.... JSON_PRETTY_PRINT while developing would be nice though, but is completely unrelated to this issue. Feel free to file a feature request against 8.1, I'd like to have that too :)

#59:

• Unpostponed #2407187: Optimize LibraryDependencyResolver::getMinimalRepresentativeSubset() and win >=4%, #2407195: Move attachment processing to services and per-type response subclasses and #2407201: Consider introducing AjaxResponse::addAttachments() (with patch) — follow-ups that were already filed in #59.
• For #2407195, I attempted a attempted patch, but ended up posting a comment explaining the roadblocks.
• Patch with AjaxResponse::addAttachments() at #2407201-3: Consider introducing AjaxResponse::addAttachments().

#65:

1. That'll be handled in #2407187: Optimize LibraryDependencyResolver::getMinimalRepresentativeSubset() and win >=4%.
2. All this cares about is whether #attached is available or not. There's no need to check for printing.
3. Agreed, did it as part of #2407201, see #2407201-5: Consider introducing AjaxResponse::addAttachments().

Patch that defaults JS to the footer: now available, and very simple, at #784626-148: Default all JS to the footer, allow asset libraries to force their JS to the header.

WHEW! That was all the follow-up stuff.