By default, don't aggregate jquery.js

pwolanin just tells me in IRC that this issue is a duplicate of #721400: Order JS files according to weight, don't change filenames for aggregated JS/CSS and we're just waiting for a commit on the other issue to continue with this discussion

subscribe.

All the counter arguments regarding this change over in #721400: Order JS files according to weight, don't change filenames for aggregated JS/CSS still stand.

This patch leads to a performance decrease for visitors that only visit one page (i.e. the very first experience with a web site).

A proper solution for this would be to build JS/CSS files differently for anonymous and authenticated users. But that's material for D8, resp. http://drupal.org/project/sf_cache

Right. So we want to penalize all sites that install any modules? That does not map to my understanding of how people build sites with Drupal.

Also, that's plain wrong: Allow anonymous users to post comments, create an article, log out, and now visit that page. I count 6 JavaScript files there.

True, by default, anonymous users cannot post comments. Since when do we optimize performance for the non-altered default configuration of Drupal core?

I don't think so. Most sites have some JS on all pages. And nowadays, due to search engines, RSS feeds, facebook/twitter/whatever statuses, the first page that most people visit first is a single content page. Modern sites allow commenting or some other kind of user interaction (voting/rating, anyone?)

Also, this is not a bug report.

Also what 6 javascript files do we need for anonymous users to post comments?!

The one disadvantage of this patch is that visitors using IE6/7 on pages with JS (say, blog comments) will see slightly reduced apparent performance on their first visit. I tried doing some IE7 profiling with Fiddler, simulating a network delay with tc, but couldn't come up with numbers that looked at all realistic (my tc-fu must be lacking). However, I did observe that with this patch, jQuery loads, then all the CSS, then the second (aggregated) script, then images. So the DOM is ready, with CSS applied, even more quickly than it is without this patch. The perceived delay will be in script behaviors applying to the page, and in image load -- not as bad as if the page itself were taking longer to load and be readable.

The advantages of the patch, on the other hand, are pretty significant:

- Those same IE6/7 visitors will see improved page load times every other time they have to load any JS - after a cache clear, or on a different page with a different set of JS.
- Everyone using a decent browser (including IE8) will enjoy the same performance improvements, and won't notice any delay on first visit.
- We could follow up with an option to use a CDN jQuery (if we can get that into core at this date in the name of performance, or in contrib if not) which would hugely speed things up. Most people would have it in browser cache, and even if they didn't, it would be coming from a different domain and therefore load in parallel.

On balance I think this is a performance win out of the box for almost every site. For sites with any significant JS, or where there are lots of content creators, it's a big performance win. I'd really like Drieschick's opinion of whether we could get a CDN jQuery option into core, because that would entirely mitigate the one disadvantage of the patch. But as it is, I think this belongs in core, in D7, so RTBC.

@pwolanin, I think ksenzee means a setting in the performance UI? Could be nifty.

Either way agreed with rtbc status on this one.

I don't think we need a generic CDN setting in core, if you set up a CDN, you can also install a module. A setting to point to a public CDN (does MS have one too?) would be a cool feature, but yes also completely a special case for jQuery somewhere.

Providing additional UI widgets to use CDN versions of JS libraries in core is definitely out of scope for Drupal 7. hook_js_alter().

Has sun's feedback been addressed? I can't tell.

I think ksenzee handled it well in #20.

Additionally I'd add that anonymous user performance is about the same between D6 and D7, whereas authenticated performance is about 25-40% worse in PHP, but that affects response times before you can even download javascript. So if there's any trade-off being made at all (and I'm not sure there really is), then it's in the right direction here.

jQuery is a special case, so I don't know why we shouldn't special-case it. Er, in contrib, I mean, of course...

Need to think some more about this one.

Clearly there are pros in terms of download pipelining, but if aggregation best practices (no conditionally loaded scripts should ever be aggregated) the total download size is the same. This is probably beneficial overall for most sites, but I think is probably far from the optimal split either ("optimal" is a very hard to determine, of course!). I wonder if there is a simple way we could split up the aggregate files (e.g. into 4 chunks, max N files/chunk, ordered by weight) up to take best advantage of the pipelining.

OK, so I have had a think. Sorry this is a little long, but there is a lot to cover.

Reading pwolanin's comments in the previous issue description, I think there are some misconceptions with how JS/CSS aggregation is supposed to work, so I will address that first.

Firstly, in core and any (no broken) contrib modules a user should only ever download 1 (yes, 1) aggregate file for each scope (print CSS, footer JS etc) for the entire site - no matter what page you visit, if you are anonymous or admin. This means you should only ever download 1 copy of jQuery. This is how it is intended to work. Probably what you have observed is that some contrib modules ignore the documentation for setting the aggregate parameter and set it on JS/CSS that is added conditionally (e.g. on certain pages only, or for certain roles only). This totally breaks aggregation, and if you have several of these bugs operating together you could very well be downloading a new large JS or CSS file on every page, which (depending on your latency and connection speed) end up slower than if aggregation was disabled. I think it is important to understand that the bug here is in these contrib modules, and that is where it should be fixed. This could be done by submitting patches back to these module maintainers, or in core by improving our documentation or changing the way JS/CSS files intended for aggregation can be added (e.g. through info files only), which I think is the most effective method, ultimately.

Secondly the original post mentions the cost of redownloading the aggregated file when a single file changes. However I don't see this as an issue - few sites update JS frequently enough to make this an issue - even the most active sites I work on make a push live at the most every 2 weeks. Most of the time, the aggregated file would have been purged from a full browser cache by that time, so you would be redownloading anyway. In addition, if sites do have regularly and rapidly changing JS (on a frequency of hours or days), it is trivial for them to simply not aggregate the files that are changing frequently, thus avoiding the problem.

In short, I think neither of the above are valid reasons to consider non-aggregating jQuery (or splitting up JS/CSS files other ways). There does seem to be another valid reason to consider this however, which is browser pipelining (described in the Steve Souders article, although that also references some specific JS loading workarounds that are definitely in contrib-land for D7).

Browser pipelining of JS is a somewhat tricky issue however, because only specific, newer browsers (IE8, Firefox 3.5+, Safari 4, Chrome 2+) support it. Any advantages we bring to these browsers are automatic losses for other browsers which will still block all other resource downloads until all JS is downloaded (and the latency of separate files over a single file of the same content size is tremendous). That said, looking at some stats, that segment of browsers adds up to ~50-65% (depending on who you ask), and considering the likely lifetime of Drupal 7, it seems that it could be worth moving in that direction. Splitting out jQuery would certainly be a simple first step in that direction, if that is a direction we want to go in. However, to get the maximum benefit of pipelining, we should look at splitting the aggregate file into files of approximately the same size. In addition 2 chunks may or may not be the optimal number of files - we would need to consider the number of available pipes in these browsers (6 for all, I believe), but also the "likely" number of non-aggregated JSs on the page (considering the latency, they need a dedicated pipe each, if possible, which could be enough to put us back to a single chunk) and finally the cost of having additional files for the older, non-pipelining browsers. Either way, unless we can figure out a very simple way of adding some flexibility here (no adding sf_cache to core!) it seems that we are probably down to "status quo" vs. "splitting out jQuery" for Drupal 7.

In other words, I think this is a "definitely maybe" - until we have more stats on our distribution of JS on typical sites, any decision here is just a random guess IMHO. Some information (which would be perfectly valid to gather from D6 sites with aggregation enabled) that I think could help determine the right answer is:
* What is the size of the aggregated file, compared with jquery.js for sites with "average" numbers of modules installed (if they are similar sizes, then good)?
* How many aggregated files are loaded on "typical" JS-active pages on "average" sites? (if 1, then good)
* How many non-aggregated files are loaded on "typical" JS-active pages on "average" sites? (if less than "6 minus number of aggregated files", then good)
* Is there a simple approach to splitting aggregate files, that would be more optimal than just splitting out jQuery, but also doable for D7? (if jQuery is not ideal to split out, and we can do this, then good)?

Setting this back to needs review until we have some information to inform this, or someone has some other convincing reason to split off jQuery.

There are also comments above about using a free public CDN for jQuery (and perhaps other library code). I think this would be advantageous enough to outweigh the potential downsides above - not only would users get blazing download speeds for it, but there is a serious possibility of it being cached (from another site) before they even come to the current site. However, I do think it needs some kind of UI control (even if it is only a $conf variable), because some users might not want to share all their visitor hit data or entrust site security with a 3rd party, and some sites might be intranet institutional or developing world sites, where outside internet access is not available. I think we need to hear from a committer what leeway there might be in terms of $conf, or I don't see how this can proceed. I also think that this discussion should be in another issue (if it is not already).

I don't believe core should be hacking around buggy contrib modules. These bugs are trivial to fix - either disable the aggregate flag or move the JS/CSS addition to hook_menu. We can also improve documentation in core, or look at ways of encouraging correct usage through API changes.

What's more, if we are starting from the point of view where aggregation is broken on a particular site, it is not clear to me at all that splitting out jQuery is necessarily an improvement, unless we are relying on pipelining (which as you can see from the above, is quite tricky to analyse). The latency costs of each http request should not be underestimated. For example testing downloading jquery.js alone (from both d.o and the Google CDN) it takes about 160ms on my connection to download it from scratch. However once it is cached the 304 cache check (which should be pretty much all latency) takes about 100ms. Hence, if you are already redownloading an aggregate file (because aggregation is broken by contrib), and can't take advantage of pipelining - you could very well be worse off with this change. The original CSS aggregation post (linked to above) has more data demonstrating the effects of latency vs. download size.

I think we should change the default in drupal_js_defaults(). For example when adding #attached['js'] I don't remember seeing any code, nor the API examples, setting that to FALSE specifically. Also I doubt module authors are deliberately setting $cache = TRUE, they're likely just not setting it either way, given the argument is optional, although I didn't grep to confirm that.

Flipping the option's default value was also my first thought. But then again, I guess that more than 95% of all JS/CSS that's loaded by modules is loaded conditionally only, i.e. for a certain form, themed element, etc. If I'm not mistaken, then only system_init() adds JS/CSS more or less unconditionally. In turn, this would mean that almost none JS/CSS would be aggregated (when just changing the default value).

The more we are discussing this, and considering that we dynamically attach JS/CSS to page elements in D7 now, the more I feel that D7's JS/CSS aggregation will create endless separate aggregates, loading a new one on almost every page.

In turn, this would mean that almost none JS/CSS would be aggregated (when just changing the default value).

If it only appears conditionally, that's the correct behaviour. If it appears every page, then it should be added in hook_init() (or .info if it's in a theme)

Ouch - yes, default to TRUE is horrible! I can't believe we just noticed that, aieeeee! The option causing us grief here is 'preprocess', by the way - not 'cache' ('cache' affects browser caching for non-aggregated JS only, not aggregation v.s. non-aggregation).

I added an issue for changing the defaults at #769226: Optimize JS/CSS aggregation for front-end performance and DX - I think this will go a very long way towards fixing this issue. Once that is resolved, let's return here to discuss the affect of pipelining and the merits of splitting out jquery, or other approaches.

@pwolanin: That's already the purpose of #769226: Optimize JS/CSS aggregation for front-end performance and DX

You patch was aggregating a bunch of things conditionally, so I would not expect it to work (they need to be moved to a global scope if you are going to set preprocess to TRUE).

Let's work on this over at #769226: Optimize JS/CSS aggregation for front-end performance and DX, where there is an existing patch - if we can't get that in (please, nooo!) or we want to revisit this from a pipelining point of view then we can return here.

It's clear that we are totally wasting bandwidth currently by generating countless of different aggregates. That's entirely contradictory to the original purpose of aggregation, because clients are loading a new, only slightly different, but full aggregate of all files on almost every page.

Therefore, postponing on the global preprocess option fix.

Subscribe.
Ability to load scripts from CDN jquery well relieve the server.
Just do not forget about the 3 networks to choose from CDN: Google (best choice), Microsoft, Yandex (faster for xUSSR countries)

Subscribe.
Ability to load scripts from CDN jquery well relieve the server.
Just do not forget about the 3 networks to choose from CDN: Google (best choice), Microsoft, Yandex (faster for xUSSR countries)

just adding some context for D8

#1541860: Reduce dependency on jQuery
#1279226: jQuery and Drupal JavaScript libraries and settings are output even when no JS is added to the page (currently jquery+drupal.js+once is always loaded, even when there is not JS)

I guess this issue is still relevant but eventual benchmarks needs to be updated to reflect what's going on and tested with IE8+ and mobile especially to see if it's still better to not aggregate jQuery.

This issue has been silent for well over a year. It boils down to "we want smarter aggregation".

While extremely informative and insightful, #31 is no longer reflective of today's reality. All CSS/JS in D8 is now added conditionally, using #attached. Yet many of this conditionally loaded CSS/JS exists on all or nearly all pages. hook_init() is no more. Conclusion: we need the aggregation system to figure out somehow which conditionally loaded CSS/JS typically co-exists, and make aggregates out of those.
I don't see any of that in this issue, so I'm closing it.