Use AdvAgg to help speed up d.o according to Google PageSpeed Insights & webpagetest.org

Created this issue #2499853: Combine CSS files by using media queries caused issues on staging.devdrupal.org

I've ID and fixed the issue with the "Combine CSS files by using media queries" option.

Patch: #2499853-3: Combine CSS files by using media queries caused issues on staging.devdrupal.org

Highly recommend https://github.com/sqmk/pecl-jsmin/. It's a lot quicker than using regex inside PHP. The admin interface should have recommended the github version as I'm assuming d.o runs on php 5.3.10+

FYI: The next group of settings require the AdvAgg Modifier module.

2015/06/04 - Here's the new baseline
http://www.webpagetest.org/result/150604_08_18E8/
2015/05/22 - Old one
http://www.webpagetest.org/result/150522_8Y_1AQQ/
Did some searching and found
2015/02/26 - Pre AdvAgg
http://www.webpagetest.org/result/150226_C8_17RN/

I'm going to guess that the next set of changes with the modifer should speed things up by about 100ms. Once we do async css I'm going to estimate that we can hit around 800ms for the start rendering line based off the repeat view; which is dependent on this issue #2396609: Inline critical css.

Another idea we can do is manually move js around via hook_js_alter in order to generate better bundles; thus increasing the browser cache hit ratio as the bundler will be able to generate more optimal groups. This change won't improve a single page view like webpagetest checks but you should see a slight speedup in the GA reports, since browsers will download less bytes as they navigate around. This has to be a manual process as D7 does not know the JS dependency tree (could be automated in D8 due to dependencies being known). I should add on an admin section to the bundler so it can advise on better ordering of CSS & JS; the data is there, just need to pull it out. Edit: created this #2500791: Have an admin section of the bundler offer ordering analysis/advice; build out the core dependencies and create a reorder function/algorithm

Looking over the latest JS and we should also check the "Move all external scripts to the top of the execution order" box as partner.googleadservices.com/gampad/google_service.js is in the middle of aggregates now.

OK I'll see what I can do. Thanks for the report :)

Deferred inline JavaScript Execution should be reasonably safe. Try that one instead and I'll see what I can do to get admanager working with footer js.

I have 3 bugs and a feature 1/2 done to do before 2.11 can go out. Tuesday is the soonest it could go out.

Feature:
#2501325: Use DNS Prefetching link tag for external CSS/JS - Create checkbox on admin page.

Bugs:
#2500803: Deferred inline JavaScript Execution breaks inline loadCSS function definition. - Have a fix in mind, need to test.
#2500203: advagg should ignore script-tags containing non-javascript content (such as underscore templates) - Need to text non xpath regex before committing; also see if xpath can be a whitelist.
#2502633: JS in footer breaks Google Ad Manager - Reported here so give it a day or so to get the fix out.

I'll be releasing AdvAgg 2.x-7.11 around noon PST on the 11th (tomorrow) unless another bug report comes in.

2.11 comes with a new feature that should help d.o

On admin/config/development/performance/advagg, 4th checkbox down has the "Use DNS Prefetch for external CSS/JS" checkbox. This has to do with the 14kb of data in 10 packets Slow-start TCP issue that is found in section 3 of this https://developers.google.com/speed/docs/insights/mobile. Because of the New Relic inline JavaScript, the DNS lookup for partner.googleadservices.com is delayed as that part of the HTML ends up below the 14kb limit.

Here's more info for slow start on centos: https://ckon.wordpress.com/2013/03/11/centos-6-4-supports-iw10-tcpip-tun...

These 3 issues should be fixed in 2.11; all having to do with google ad manager.

Move JS to the footer - All. Conflicts with google_admanager.module.
Deferred JavaScript Execution: Add The defer Tag To All Script Tags
Deferred inline JavaScript Execution: Put a wrapper around inline JS so it runs from a setTimeout call.

Crossed off 2.11 and moved the 3 items that got fixed in the latest version of advagg out of the needs work section.

I've been looking at the Connection View from the last webpagetest run and right now it seems like we should reduce the number of JS bundles from 4 to 3 as we create a new connection and it only downloads 1 thing, a js aggregate. This will be the final step we do so we can hold off on it for now as the jsmin, footer, and defer might change how many connections to d.o are created.

Also looking at this and there are 2 PNG files that take forever to get
https://www.drupal.org/files/styles/case-4-2x/public/Drupal-case-study-1...
https://www.drupal.org/files/styles/case-4-2x/public/Case-study-image2.p...
https://pngquant.org/ might be a good solution for these 2 files; will need to see if the drop from 24bit to 8bit causes noticeable banding. If you put https://www.drupal.org/modules/image/sample.png through pngquant it always looks awful.

Uploaded the 8bit versions of the PNGs. I don't see any banding issues.

When using a slow 3G connection only 5 connections are used for d.o
http://www.webpagetest.org/result/150616_KK_1854/1/details/
When using a fast cable connection all 6 are used and it looks like 2 connections are now only used for JS
http://www.webpagetest.org/result/150616_0E_187Z/1/details/
Found it interesting that the connect speed changed this.

Looking at the DNS lookup times; anyway someone can tell me what bit of JS hits these domains?

securepubads.g.doubleclick.net
ssl.google-analytics.com
pagead2.googlesyndication.com
tpc.googlesyndication.com
tag.perfectaudience.com
pixel-geo.prfct.co
secure.adnxs.com
analytics.twitter.com
ads.yahoo.com
www.facebook.com
p.univide.com
cs.marinsm.com
pixel.prfct.co
ib.adnxs.com

From my local testing I've added these into the latest advagg dev

    // Google Ad Manager.
    if (strpos($value['data'], '/google_service.') !== FALSE) {
      if (!empty($value['dns_prefetch']) && is_string($value['dns_prefetch'])) {
        $temp = $value['dns_prefetch'];
        unset($value['dns_prefetch']);
        $value['dns_prefetch'] = array($temp);
      }
      $value['dns_prefetch'][] = 'https://cm.g.doubleclick.net/';
      $value['dns_prefetch'][] = 'https://pagead2.googlesyndication.com/';
      $value['dns_prefetch'][] = 'https://partner.googleadservices.com/';
      $value['dns_prefetch'][] = 'https://pubads.g.doubleclick.net/';
    }

    // Google Analytics.
    if (strpos($value['data'], 'GoogleAnalyticsObject') !== FALSE) {
      if (!empty($value['dns_prefetch']) && is_string($value['dns_prefetch'])) {
        $temp = $value['dns_prefetch'];
        unset($value['dns_prefetch']);
        $value['dns_prefetch'] = array($temp);
      }
      $value['dns_prefetch'][] = 'https://www.google-analytics.com';
      $value['dns_prefetch'][] = 'https://stats.g.doubleclick.net';
    }

Looking at this and there must be some settings for GAM and GA that control what domains get hit as ssl.google-analytics.com & securepubads.g.doubleclick.net are related to these 2 modules.

https://developers.google.com/analytics/devguides/collection/protocol/v1...

Been playing around with the settings for advagg and I've updated the todo list with the latest changes from advagg. I'll roll 2.12 on monday if the issues are pretty quiet over the weekend
https://www.drupal.org/project/issues/advagg?version=7.x

See anything broken here? https://ci-drupal.redesign.devdrupal.org/

AdvAgg 7.x-2.12 is out.

Changed to 2.13 as some bug reports came in today that needed to go out.

Just released AdvAgg 7.x-2.14. Fixes a hard to repo issue (race condition) with defer js and the opera browser.

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%...
Now reports that the google ads js is blocking and drupal css is blocking. When not using google ads the score is a lot better as the blocking js is gone
https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%...

So js defer is doing it's job!

Thanks again for releasing all these changes out :) Is there an ETA for when d.o will move to php 5.3.10+?

Any reports on the before and after frontend performance differences from google analytics and new relic? Always good to verify things are moving in the right direction. I'll play around with connection speeds and see what the optimal bundle size is; 4 is a pretty good number so it might not change.

Thanks for the PDF. Looks like it only tracks when the page is ready and not when it starts to render and thus starts to become available for user interaction. Thus the only way to improve the Google Analytics metric would be to make the page less complex.

I've ran some tests and here are the results with the start render time being shown below.

d.o homepage: https://www.drupal.org/
Native - Start Render: 0.691s
Cable - Start Render: 1.695s
3G - Start Render: 4.390s
Edge - Start Render: 31.991s

d.o project page: https://www.drupal.org/project/drupal
Native - Start Render: 0.495s
Cable - Start Render: 0.588s
3G - Start Render: 2.290s
Edge - Start Render: 7.992s

When looking at both of these the google page speed advice is pretty spot on from #57. For the homepage the google ad javascript is blocking the page from starting to render. for the project page the css files are blocking the page from starting to render. The faster the connection the less of an issue this is. Once things start to slow down every file that blocks the rendering really starts to make it painfully slow.

#2396609: Inline critical css will speed up the project page and just about any other page that doesn't use google ads (most of d.o). For slow connections like Edge this would start the render around 5.5 seconds (right after the HTML has been downloaded); an improvement of 2.5 seconds of where it's at now. On 3G it would start around 1.5 seconds. The homepage is blocked by the google ad js so my guess is that inlining css will not help on the front page.

Browsers like to use 2 connections for CSS so we could drop the bundles from 4 to 2 in this case; will give us a worse browser cache hit rate when browsing around but it would speed up tests from webpagetest. Overall I think this would be good to try.

JavaScript doesn't seem to have this limitation so I would try 6 or even 8 bundles now that the js is not blocking on most pages. If the homepage is very important then I would keep the bundles below 6 for JS as the blocking ad JavaScript really messes up the benefit from defer. Also noted that if the CSS is non blocking then we might want to increase the bundles for CSS to get a better browser cache hit ratio for CSS.

Right now the next recommendations is to move the css bundles from 4 to 2 and js from 4 to 5.

Project Page
3G Start Render: 1.990s
Edge Start Render: 5.598s
Noticeable improvement

Home page
3G Start Render: 5.295s
Edge Start Render: 15.894s
No improvement due to the blocking ad javascript.

Moved the "connection view from webpagetest.org" item to the done section.

Will investigate if Google Admanager can be lazy loaded. Will also see if Google Analytics "Locally cache tracking code file" helps speed up the page load time.

Looks like google supports async ads; the https://www.drupal.org/project/google_admanager module does not.

If you go here https://support.google.com/dfp_sb/answer/6033499?hl=en and select async iframe it should generate the correct tags. From here I can see what can be done with the module. The code should look something close to this
https://support.google.com/dfp_sb/answer/1651549#async

In terms of speeding up the total page load time I'll be trying to write the async tag instead of using a shim for google analytics. On the project page that holds up the the "Document Complete" timer. I have this working locally, will test on our sites as well as https://ci-drupal.redesign.devdrupal.org/ tomorrow. In the process of fixing the GA 1.4 issue I addressed this as well.

Test results from https://ci-drupal.redesign.devdrupal.org/project/drupal
Doc Complete 6.431s - baseline
Doc Complete 5.603s - Move GA analytics.js code from inline to be a file
Doc Complete 5.229s - Move GA + Async JS Execution: Group together in the header

Looks like not using js to write the js async tag for GA makes the page load quicker since there isn't a lot of content on the page to get; get a slight improvement on top of it if moving the GA js code to the header since it loads other things after the GA code has ran.

According to https://support.google.com/analytics/answer/2383341?hl=en this change should improve the Avg. Page Load Time (load completion) reported in google analytics.

Doc Complete 5.078s - Move GA + Prefetch stats.g.doubleclick.net/robots.txt

Moving the async js to the header caused the first js file to not get downloaded till later; not ideal, the Start Render time is slower. By prefetching stats.g.doubleclick.net/robots.txt the connection to stats.g.doubleclick.net is open and thus the request for stats.g.doubleclick.net/r/collect is sped up quite a bit since the Initial Connection & SSL Negotiation where already done and this doesn't seem to effect the Start Render time since the order of what gets downloaded hasn't changed.

So using the latest dev my next recommendation is on the admin/config/development/performance/advagg/mod page; under "JS" -> "Optimize JavaScript Ordering" -> "Experimental Settings" check both boxes, " Move Google Analytics analytics.js code from inline to be a file" and "Prefetch stats.g.doubleclick.net/robots.txt".

Noted that we're using both of these options on our production site without GA issues.

Edit:
Here's a advagg disabled, core enabled snapshot of the project/drupal page.
Start Render 1.497s; Doc Complete 1.796s Native
Start Render 2.088s; Doc Complete 2.380s Cable
Start Render 4.890s; Doc Complete 7.201s 3G
Start Render 14.888s; Doc Complete 21.808s Edge

Advagg enabled
Start Render 1.490s; Doc Complete 1.830s Native
Start Render 1.588s; Doc Complete 2.002s Cable
Start Render 3.488s; Doc Complete 5.127s 3G
Start Render 7.690s; Doc Complete 14.181s Edge

Made a graph of the data

Heads up that rupl created an excellent guide on using critical css; the only downside is it's labor intensive and any change to css would require a regeneration of all the critical css from grunt. From what I can see this would give drupal.org a 25% improvement in the start render time for all connection types except for native, that seems to be CPU bound on webpagetest. Still not ready for prime time but good to know where it's at.

Thoughts on #66 in regards to async google ads?

https://www.drupal.org/project/drupal
Before
3G doc complete 4.9s
After
3G doc complete 3.5s

I'd guess that this speedup would be noticed in the GA tracking in the next couple of days.

#66 was in regards to the homepage ad and how it blocks the rendering.

I'm getting bad inline js for the drupal.settings only on the project usage pages: https://www.drupal.org/project/usage/drupal

Getting junk like this:

<!--//--><![CDATA[//><!--
function init_drupal_core_settings() {jQuery.extend(Drupal.settings,{"basePath":"\/","pathPrefix":"","ajaxPageState":{"theme":"bluecheese","theme_token":"DvLr_ellBbPFtVMxMYDU4qtnkcE9n2r0SF3Yf0CxT0w"},"jcarousel":{"ajaxPath":"\/jcarousel\/ajax\/views"},"googleanalytics":{"trackOutbound":1,"trackMailto":1,"trackDownload":1,"trackDownloadExtensions":"7z|aac|arc|arj|asf|asx|avi|bin|csv|doc(x|m)?|dot(x|m)?|exe|flv|gif|gz|gzip|hqx|jar|jpe?g|js|mp(2|3|4|e?g)|mov(ie)?|msi|msp|pdf|phps|png|ppt(x|m)?|pot(x|m)?|pps(x|m)?|ppam|sld(x|m)?|thmx|qtm?|ra(m|r)?|sea|sit|tar|tgz|torrent|txt|wav|wma|wmv|wpd|xls(x|m|b)?|xlt(x|m)|xlam|xml|z|zip"},"urlIsAjaxTrusted":{"\/project\/usage\/drupal":true}});
  

      // Set this to 100 so that this function only runs once.
      advagg_mod_3.count = 100;
    }
  }
  catch(e) {
    if (advagg_mod_3.count >= 40) {
      // Throw the exception if this still fails after running 40 times.
      throw e;
    }
    else {
      // Try again in 250 ms.
      window.setTimeout(advagg_mod_3, 250);
    }
  }
}
function advagg_mod_3_check() {
  if (window.jQuery && window.Drupal && window.Drupal.settings) {
    advagg_mod_3();
  }
  else {
    window.setTimeout(advagg_mod_3_check, 250);
  }
}
advagg_mod_3_check();
//--><!]]>

My guess is the regex is barfing big time here; only happens on these pages as well. Quick recommendation is to turn off "Deferred inline JavaScript Execution: Put a wrapper around inline JS so it runs from a setTimeout call" until the bug can be fixed.

Looking into this more and I think it's the render cache; Trying to pinpoint the issue but turning the caching from "Aggressive Render Cache" to "Normal" seems to fix it on https://ci-drupal.redesign.devdrupal.org

Committed: #2558271: Render cache and Flot module

Looks like this has been an issue for sometime; as this code hasn't changed for a little while. Never was reported because the chart still works.

@drumm
Any chance I could get another pdf of google analytics, like what's in #61? Even better if you can filter out the homepage as the ads prevent further optimizations.

Thanks for getting the data. Most interesting difference between the two is the tail end of the browsers average page load time. UC Browser went from 13.3 down to 5.4 seconds. The main graph doesn't seem to show the speedup, my guess is most developers have a fast connection so the average doesn't move because of that. Most of these optimizations only show improvements when the connection speeds are slow.

Can you geo filter? Pick a region that is far away from the d.o server?

Edit: looks like the UC Browser is a pretty good filter for slow mobile connections https://en.wikipedia.org/wiki/UC_Browser

With a huge user base in China, strong adoption in India and continued growth in emerging regional markets, UC Browser reached 500 million global users in March 2014

@iflexion
The start render time is pretty bad on that site, 4.7 seconds before anything shows up and 6.7 seconds until the site is "usable" (important text can be read) when on a 3G connection. The css is still render blocking and now it's not getting downloaded first, thus pushing back the start render time. PageSpeed is deferring the js correctly so that is helping you out. The downloadable fonts are blocking the text from displaying thus the 6.7 seconds until the site is usable statement.

I'm not sure what point you're trying to make, AdvAgg is also open source, but it's built with Drupal in mind and thus can usually get better results than a tool that works with just about anything like PageSpeed. https://fourword.fourkitchens.com/article/use-grunt-and-advagg-inline-cr... shows how to get the start render time down by making the css non blocking; in the case of your site it should make your start render time be in the sub 1.5 second range on a 3G connection. You'd need to install the advagg font module as well so downloaded fonts don't block the text from displaying.

Heads up that the recent change to d.o killed front end performance. Looks like sync loading of optimizely js.

Before at 3G Speed - Start Render: 1.8s, Document Complete 3.5s

After at 3G Speed - Start Render: 4.4s, Document Complete 6.6s

Thanks for removing the A/B code. I've created this issue in advagg so the dns-prefetch will work in the future if optimizely is used again #2612134: Allow for DNS prefetch to be at the top of the meta headers.

http://www.webpagetest.org/result/151116_DT_1GJW/ - Start Render 3.0s, Document Complete 4.9s.

Looking into the details of this and it appears that the previously used webpagetest server (First Byte Time Grade: F) isn't close to a Fastly CDN. Picking one that is closer (SF) gives better results (First Byte Time Grade: B).

http://www.webpagetest.org/result/151117_EP_8P/ - Start Render 2.6s, Document Complete 4.2s.

Looking into why the start render time is a lot slower in comparison to the previous run from August, I'll take a good look at the 2 css files
Aug 28th css files:
css__AVcEcTVLBLX8vW4CyfA90b53b4fE0NaYtvcPwowjREk__BTnu98PCe61qizCcD6wWemE_JSPA6B5xt9Bk84RTPzA__YnARITFFciBSl3CVzI-UH0Ia2zJaJD5zrWNkdXgi_JU.css - 3.0kb
css__9PvYrtmvYA3RUchwLSbpxZK_SFjh7-JXqavUE0B5ldI__Kl-q_AwS8K1oloJbwp478b5mVE6_mkHYsF8BhFbYRKg__YnARITFFciBSl3CVzI-UH0Ia2zJaJD5zrWNkdXgi_JU.css - 21.8kb

Nov 16th css files:
css__AVcEcTVLBLX8vW4CyfA90b53b4fE0NaYtvcPwowjREk__BTnu98PCe61qizCcD6wWemE_JSPA6B5xt9Bk84RTPzA__OpNX9-nIOED3tKWoeE--Vb5COV1cZHtanu2Cr3kvovc.css - 3.5kb
css__h02uXvYwovBwCRZ9iuLn292GJD1JiuXGUdEX9ZPxYLM__NsY08RzWBltwnp402XKFu4btr2Bxi0I0pCCsfr5CzhY__OpNX9-nIOED3tKWoeE--Vb5COV1cZHtanu2Cr3kvovc.css - 22.7kb

Couple of interesting things. The first file css from Aug and the first file from Nov include the same group of files (first hash) and have the same content (second hash) but according to webpagetest it took an extra 0.5kb. My guess is the extra traffic is due to more headers getting transferred. The second css file is different on all accounts and I'd guess it grew 0.4kb in size. My guess is that this combination with the 0.5kb in headers and 0.4kb in content makes that file take 3x as long to download due to Slow-start TCP; as in the filesize hit a magic number that all of a sudden make the file a lot slower to download. If both css files came in at roughly the same time by having the a similar size in kb the start render time would be right around the 2 second mark. I created an issue to look into solutions for this #2616834: Allow for the bundler to optimize for more even filesize vs file count.

Both issues have been taken care of. Would like to test on ci-drupal.redesign.devdrupal.org but it appears that is no longer accessible. What should I use to test this out?

New theme is about the same. Still looking like the css files could be better balanced when it comes to filesize (4.3kb vs 23kb).

http://www.webpagetest.org/result/151119_8P_15ER/ - Start Render 2.3s, Document Complete 4.2s.

Played around with this on test and https://www.drupal.org/sites/all/themes/bluecheese/css/styles.css is by far the biggest file and it will end up being its own aggregate. It should end up right around the 20kb mark so this new change should help with Start Render time. It won't help if fastly has a low TCP slow start window value; this post gives me hope that fastly has a good default: https://www.fastly.com/blog/jason-cook-linux-conference-australia-2014

I got enough changes for a new release of advagg next week so I'll be working on getting that out the door. Once that is done I'll ping this issue for the upgrade info and the new settings to use. Hoping for around a half a second improvement for the start render and document complete times.

New version of AdvAgg was released today; recommend upgrading to it and changing the bundler grouping logic to be filesize. Also a new submodule included in the release that might be of interest is Subresource Integrity.

Looks like we hit the wall in term of getting the most performance out of the current setup. Results are pretty close to what we had before.
http://www.webpagetest.org/result/151211_97_Z19/ - Start Render 2.6s, Document Complete 4.6s.

Guess the final steps is jsmin with php 5.3.10 or newer, inlining critical css and http 2 #2046731: HTTP/2 on drupal.org (was: SPDY on d.o)

@drumm
Is it true that d.o is now off of PHP 5.3? If so we should get jsmin going. Side note that I'll be getting AdvAgg 2.17 out the door sometime this week so should probably wait for the update before turning on jsmin.

not really, we just did
pecl install jsmin
as instructed here https://github.com/sqmk/pecl-jsmin

Looks like the 2.x version of jsmin has issues with the systems locale setting (#2627468: JsMin: Browser console gives errors like SyntaxError: illegal character). So using the 1.1.0 version is preferred.
pecl install jsmin-1.1.0
AdvAgg 7.x-2.18 is out and should be ready to go as long as you don't use JSqueeze for js compression.

Any idea on when jsmin can be installed? Once again the 1.1.0 version is the one to use.
https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%...

Decided to test this out again using "Mobile Edge" as the speed limit. http://www.webpagetest.org/result/161114_E9_NY9/

The new d.o theme uses an external font "Ubuntu" which causes all sorts of trouble. Elimination of this would be ideal. If this is a no-go I can come up with a solution that would involve copying the contents of https://fonts.googleapis.com/css?family=Ubuntu:700,400,300 instead of using @import url(https://fonts.googleapis.com/css?family=Ubuntu:700,400,300);. Going either way (elimination or no import) should drop the start render time from 9.9s down to about 6.5s. If the external font is kept then using the AdvAgg Font module might be a good idea; if the external font is removed then we should expect to see about a 1 second drop in the fully loaded time as well.

Also the next version of AdvAgg came out recently; if upgrading to this I'd also Check "Preconnect" under Resource Hints. In theory this will eliminate the Initial Connection & SSL Negotiation times if the browser supports it http://caniuse.com/#feat=link-rel-preconnect. Also support for br compression was added; can be unlocked by using https://github.com/kjdev/php-ext-brotli

Created this #2827137: Inline @import statements

Did a quick check and it seems like gzip dropped off with the new version of AdvAgg in this case. The .htaccess files where changed in advagg_css and advagg_js in order to support brotli compression. I have a guess that adding this to the top of the htaccess file will fix it.

Options +FollowSymLinks
RewriteEngine on

If not removing everything between <IfModule mod_headers.c> and </IfModule> will for sure fix it. Current issue on this #2825530: gzip is failing for js / css files

http://www.webpagetest.org/result/161117_08_4E9A/

gzip is looking good now, thanks! Start render is 8.497s, down from 9.905s thanks to the Preconnect header. Still working on the advagg_relocate module; not an easy drop-in due to different font file support in different browsers; this makes Sass compilation most likely a non starter. If mod_pagespeed is being used then this filter https://developers.google.com/speed/pagespeed/module/filter-css-inline-g... should work if using the standard google link tag.

Good job on getting this done! #2046731: HTTP/2 on drupal.org (was: SPDY on d.o)

Looks like browsers are bad at allocating bandwidth for making connections to blocking resources on other domains. https://www.webpagetest.org/result/161214_XQ_BHDR/ Start render went from 8.5s (#113) up to 11.5s; makes the payoff from #2827137: Inline @import statements even better as that would remove the connection to fonts.googleapis.com giving us a start render time under 6s. Before H2 the best we could have gotten was right around 6.5s.

With the change to H2 We could bump the number of CSS aggregates created by the bundler to 4; but in reality it could be 15 for both css and js. I just wouldn't go that high due to H2 support not being good in IE or Safari http://caniuse.com/#feat=http2

AdvAgg has support for sending out the HTTP/2 Push headers but it's wastes a lot of bandwidth on repeat visitors. I'd wonder if this can be used to make the browser better allocate resources for the external connections. Under the Preload link http headers section only enable CSS and have only external checked; that might help but it'd be browser dependent.

AdvAgg 7.x-2.20 is out. Has a new module called relocate; it will inline external css files from fonts.googleapis.com currently. Targets @import as well as external css. IE8 and lower is not supported by default.

Also ships with http://bitsup.blogspot.de/2016/05/cache-control-immutable.html turned on by default. Currently support in Firefox only https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#...

Yep see the issue on my side as well... Didn't test the no httprl code path enough. Working on a fix.

Fix is committed: #2842492-3: Throws errors if httprl is not installed
Only commit to dev so far if you wanted to just grab that: http://cgit.drupalcode.org/advagg/log/

AdvAgg 7.x-2.21 is out
https://www.drupal.org/project/advagg/releases/7.x-2.21

I'm working on this issue #2862336: Look in themes/THEME_NAME/all/critical-css/urls/page_name for css files to inline
Idea being that you place the critical css in a folder inside the theme, AdvAgg see it and does the correct thing. This would make the long standing goal of having inline critical css be a lot easier.

AdvAgg 7.x-2.25 is out
Update and install the AdvAgg Relocate; it will inline css from fonts.googleapis.com currently. This will help with the start render time https://www.webpagetest.org/result/170617_MC_3DQ/.

Go to admin/config/development/performance/advagg/relocate and make sure "Enable Inlining CSS Import Statements" and "Enable Inlining External CSS Files" are both checked (defaults).

Yes that is correct. I've never seen it cause any issues so I've never removed it. I've created this issue to address your concerns #2889776: Remove no longer needed dns-prefetch and preconnect link tags

Results from today
Before: https://www.webpagetest.org/result/170626_NQ_1HK4/
After: https://www.webpagetest.org/result/170626_S1_1K3C/
Looks like a ~2 second improvement on the start render time! Also a ~1 second improvement on the doc complete time as well.

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%...
Shows what's next on the list. Gzip svg and longer browser caching of said svg files would probably be the easiest to do. Minify JavaScript is a little bit harder to pull off; any word on when the JSMin PHP extension will be enabled? And finally Eliminate render-blocking CSS is the hardest to pull off but offers the best improvement in terms of the Google Page Speed score. There is a free tool for creating the critical css https://www.sitelocity.com/critical-path-css-generator but that's not ideal if the css changes with any sort of frequency.

One week seems to be the minimum threshold for the browser cache. Default core apache rules is 2 weeks.

Looking over everything and I see a way to get a 100/100 on https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%... once I get #2862336: Look in themes/THEME_NAME/all/critical-css/urls/page_name for css files to inline better tested and then committed.

To get the perfect 100/100 would require svg compression and longer caches; also wonder why https://www.drupal.org/sites/all/themes/bluecheese/images/bg-blue-bluepr... is not cached for more than 24hrs. This would also require jsmin to be enabled but I think I figured out the issue some people where having with it, non UTF-8 encoded file like UTF-16LE. If I can get a dev enviroment for testing advagg I should be able to get the critical css better tested based on the node type or page.
https://www.drupal.org/documentation
https://www.drupal.org/project/drupal
https://www.drupal.org/8
What does Google Analytics say are the top 10 landing pages for d.o? Targeting the top 10 for inline critical css is a good starting point.

Yes that is current and correct. Thanks for the list; It'll help a lot for just picking the important critical css to inline.

/
/download
/dashboard
/docs/7/*
/project/*

Looks like /dashboard might take some minor work as its a user page and not a node page. The rest of these look like a node type page or the frontpage. Seeing how much they very in the critical CSS will be part of what I'll look at, hopefully very little.

Was looking for https://www.drupal.org/drupalorg/docs/build/development-environments/dev...
Looks like I get a permission denied (publickey) error when trying to SSH in.

A quick check in https://www.sitelocity.com/critical-path-css-generator of the 10 paths shows that the CSS has some minor changes for the documentation node; nothing major so combining these should be ok as I did here. Also did the same for projects form and node project. The need to be logged in for /dashboard to work means it'll be hard to do with a semi automated tool.

Looking over the different CSS files and it seems like a base file could be pretty useful as a lot of the CSS needed to render above the fold content is the same; this can then be used for pages like /dashboard that would be hard to do well. This is something that I haven't considered but seems like it might be useful if done well.

I'm out of town this week (post Eclipse vacation), I'll get back to you next week.

I can access the server.

Still working on the npm/yarn backend part of critical css so I haven't had a chance to get the dev.d.o really going. Is d.o debian or redhat based?

Good news is with the latest version of AdvAgg js minification can be done via drush/cron/batch-api (drush advagg-jsmin) so it can be pre-computed and then turned on so the chance of a page load stall due to js being minified is now zero; so upgrading to the 7.x-2.28 release would allow for this to be used as the jsmin php extension isn't a requirement now. If using elysia cron it will auto configure to run every 2 minutes.

Steps to install critical. I've tested these on a fresh VM of 16.04, but that doesn't guarantee it'll work.

# Get latest Node. See https://github.com/nodesource/distributions
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -

# Get requirements for chrome and nodejs.
sudo apt-get install -y libnss3 fontconfig ttf-liberation fonts-liberation nodejs build-essential libxss1 libappindicator1 libindicator7 git mc 

# Get latest headless chrome.
# https://askubuntu.com/questions/79280/how-to-install-chrome-browser-properly-via-command-line
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb && sudo dpkg -i google-chrome*.deb && sudo apt-get install -y -f && sudo dpkg -i google-chrome*.deb

# Get yarn, penthouse and critical
sudo npm install -g yarn && yarn add https://github.com/pocketjoso/penthouse && yarn global add https://github.com/addyosmani/critical.git

# Test output
.yarn/bin/critical https://www.drupal.org/ --ignore @font-face

With critical running locally getting inline critical css is one step closer. What are the steps needed to get this going?

@drumm
Only needed on dev and/or staging where the new css files can be checked in. As long as the page looks really really close to production (same class and ids) it should be good.

AdvAgg 7.x-2.29 is out with drush advagg-jsmin support

    'examples' => array(
      'drush advagg-jsmin' => dt('Minify only the files that need to be done.'),
      'drush advagg-jsmin all' => dt('Minify all js files again.'),
      'drush advagg-jsmin misc/jquery.once.js' => dt('Minify the misc/jquery.once.js file.'),
    ),

This should allow for js minification to be used since it can be pre-computed (gets put into the cache_advagg_info bin).

• Run `drush advagg-jsmin` after deployments to production (AdvAgg 2.29) and enable jsmin
• Under Relocate Check "Inline @import CSS font files in local .css files"
• Under Relocate Check "Move inline google analytics inline analytics.js loader code to drupal_add_js"
• Update nginx rules allowing for a 1 week cache of svg and jpg files under sites/all/*
• Update nginx rules to gzip svg files
• Get https://github.com/addyosmani/critical/ up and running in a dev/staging environment

Also of note https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%... is complaining about a 5+ second page load time

http://www.webpagetest.org/result/171214_AE_340cd22525516ae420b3f625c23f...
The google font css is blocking the start render; it adds 5 seconds to the start render time. Was this turned off?

2.x-2.30 released. Issue with jsmin, deferred js, and jquery.

Heads up that Google updated it's PageSpeed Insites tools
https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%...

Mobile: 3.4s FCP, 3.7s DCL, 60/100
Desktop: 1.5s FCP, 1.8s DCL, 80/100
First Contentful Paint (FCP) measures when a user sees a visual response from the page
DOM Content Loaded (DCL) measures when HTML document has been loaded and parsed

Thinking we should optimize for HTTP2 as almost all browsers support it now. Will update the todo later on this week.

Updated top level recommendations. Plan on adding some features and releasing 7.x-2.32 in the near future; recommend waiting till 2.32 is out to do any AdvAgg changes.

The work on the nginx rule and apache rule could be done now though.

AdvAgg 2.32 is out and with it comes a UI for configuring Critical CSS. I've updated the top but here's the current recommendations for AdvAgg -

• Update to AdvAgg 2.32
• Select jsmin+ on the JS Compression page
• Run `drush advagg-jsmin` after deployments to production; or run the batch operation from the UI
• Switch most AdvAgg settings to recommended
• Set the AdvAgg bundler to HTTP 2 settings
• Enable AdvAgg Critical CSS module
• Add Critical CSS via the Critical CSS page

https://groups.drupal.org/node/517292 has an overview of all the steps.

Other things that would be nice to have.

• Update nginx rules allowing for a 1 week cache of svg and jpg files under sites/all/themes/* as well as files/drupal-wordmark.svg
• Update apache rules to gzip svg files (see https://stackoverflow.com/a/21533084/125684)

Back from vacation today. Looks like I could use a test environment again.

Yep login still works! I'll take a look at it tomorrow, and thanks again for the support here.

No way to run https://developers.google.com/speed/pagespeed/insights/ on a site that uses basic auth so I did most of the tuning via what I saw when running it in https://www.webpagetest.org/

I've committed some code changes to advagg as a result of some small issues I encountered when working on the homepage. That being said I'll be targeting the landing pages listed here https://www.drupal.org/project/infrastructure/issues/2493801#comment-122... in the following days.

You can see all changes here if you're curious https://advagg-drupal.dev.devdrupal.org/admin/config/development/perform... but these are the files that are preloaded

files/cta/graphic/drupal-wordmark.svg
files/cta/background/bg-nyc.jpg
https://i.ytimg.com/vi/iHpPGGpxBEA/sddefault.jpg
sites/all/themes/bluecheese/images/icon-w-drupal.svg
sites/all/themes/bluecheese/images/icon-w-search.svg
sites/all/themes/bluecheese/images/icon-w-user.svg
https://www.youtube.com/yts/jsbin/player-vfllqtOs7/en_US/base.js

And these are the domains that get looked up

i.ytimg.com
fonts.gstatic.com
fonts.gstatic.com#crossorigin
connect.facebook.net
static.ads-twitter.com
snap.licdn.com
googleads.g.doubleclick.net
ss.symcb.com
www.google.com
static.doubleclick.net
www.facebook.com
pixel-geo.prfct.co
t.co
secure.adnxs.com
analytics.twitter.com
cs.marinsm.com
ads.yahoo.com
p.univide.com
us-u.openx.net
image2.pubmatic.com
pixel.prfct.co
cm.g.doubleclick.net
pixel.rubiconproject.com
cw.addthis.com
ib.adnxs.com
www.linkedin.com
px.ads.linkedin.com
dc.ads.linkedin.com
www.drupal.org
tag.perfectaudience.com

A lot of these domains have to do with advertising.

These can be deployed as a database change or as 3 files in the theme directory. The critical css is for this dev page with the red bar, so it can't be used for production; used https://chrome.google.com/webstore/detail/critical-style-snapshot/gkoeff... to generate it so it's not 100% perfect as well; firefox jumps around a little bit on start render due to the tool being chrome; https://www.sitelocity.com/critical-path-css-generator usually has better results. Also for the homepage I added in this inline css change to make the youtube video thumbnail show up a lot sooner

.cta-text .field-name-field-cta-body-2 .field-items .even {
  background:url("https://i.ytimg.com/vi/iHpPGGpxBEA/sddefault.jpg") center center no-repeat;
  width: 440px;
  height: 248px;
  background-size: 440px;
}

There's still no good way to auto generate the critical css at this point but doing it by hand doesn't take too long to do.

Here's how it renders on a 3G connection with advagg (most of the loading time is waiting for the youtube iframe to load)
https://www.webpagetest.org/video/view.php?id=180314_3M_da524d191919d20c...
And without but with core aggregation turned on
https://www.webpagetest.org/video/view.php?id=180314_SA_7e6133ef13eb545b...

Looks like if the #drupalorg-site-status parts of the css are removed then this will pretty much look like production if I'm not mistaken.

I added the other ones listed in #133 to the admin/config/development/performance/advagg/critical-css page. Currently only the front page has been heavily optimized with custom inline css and filling out the Hostnames and Preload lists. Doing the other pages takes about 2 minutes each as they don't need to be crazy optimized like the front page usually does on most sites. These changes can be done as a configuration change in the custom db table or as a file in the theme dir; the UI gives target filenames to make exporting easier.

If the URL is an alias (not node/2) then adding on a query string to the "Value to Lookup" section is how to test this in production without exposing it to the world. If I have time I'll probably be bugging you guys on slack tomorrow to go over any concerns and if it looks good I'll push out a new release after doing some more testing on my side.

AdvAgg 7.x-2.33 is out; only change from Friday was to the readme. You can see on the dev server how I have everything configured; all the sub modules use recommended settings.

Adding in https://github.com/bramstein/fontfaceobserver to the libraries is one thing I haven’t done yet.

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%...

Moving in the right direction!

Before #152
Mobile: 3.4s FCP, 3.7s DCL, 60/100
Desktop: 1.5s FCP, 1.8s DCL, 80/100

After
Mobile: 2.6s FCP, 2.9s DCL, 74/100
Desktop: 1.3s FCP, 1.7s DCL, 85/100

First Contentful Paint (FCP) measures when a user sees a visual response from the page
DOM Content Loaded (DCL) measures when HTML document has been loaded and parsed

Got relocate working for all the scripts running in https://cgit.drupalcode.org/drupalorg/tree/drupalorg/js/audience-extensi...

Google ads do not work if the ad has css attributes applied to it via inline critical css; something to be aware of.
#2957428: Allow for certain css selectors to not have any critical css
Other issues I'll be working on that should get us closer to having inline css be a smooth process.
#2954723: Allow for critical css to be auto generated via service (local or remote) on detected css change; add drush support as well.
#2954718: Allow for critical css to select file or db if both are present via UI.