Add GZip compression

Hi Rod, just attatch the patch to a comment and set the issue status to "patch (code needs review)"...

This is a great idea!

Please be sure to make this optional (we need a setting to enable/disable the PHP-based gzip encoding). Many people (myself included) use apache's native gzip compression module so this would might just increase the PHP workload. (I'm not sure which is more efficient, but that's irrelevant - servers with built-in compression shouldn't have to recompress stuff compressed via PHP.)

Its a great idea indeed Rod, waiting for the patch now :)

http://kaioa.com/node/78

I've just read the article at http://kaioa.com/node/78 - this seems to offer probably the best solution for what I've been looking for for many months.

If someone would attach a .zip file of Javascript_aggregator for Drupal 5 (with brief Readme instructions), I'll immediately test it on my live site.

thanks!

Here is a patch (for 5, but should be easy to reroll for 6) that adds the option to additionally create gzipped versions of the aggregated file in the style of the kaioa.com HOWTO. What is missing is instructions (in the handbook page for this module) explaining how to add the lines to .htaccess. If someone wants to write these up I would be happy to reroll the patch with a link to the docs from the field description on the form.

I tried #3. I think I got it working but not so sure. In /sites/mysite.com/files/css I do have fa36521fb0cdee3d2930ebee70f67c0f.css (45KB) and fa36521fb0cdee3d2930ebee70f67c0f.css.gz (10KB).

however, when use Live Search Webmaster Center i got "HTTP Compression not enabled".
What exactly that means?

@najibx - as I mentioned, you need to add the sections to your .htaccess file as described in http://kaioa.com/node/78

Awesome work, Owen! It would be wicked to automatically create the .htaccess file in the /js/ folder if it doesn't exist. Oh man, this is cool.... I'll have a hack at it on Monday for both the Drupal 5 and 6 version if no one else has a try.

Hey - I like the idea of creating a separate .htaccess just in the /js/ folder - that would resolve one of the issues people had with this approach in core - that it adds additional rewrite rules. These would now not be triggered unless you were in the right place already :)

I sense a core patch coming on...

Committed to both Drupal 5 and 6 branches:

http://drupal.org/cvs?commit=159063
http://drupal.org/cvs?commit=159051

Testing worked here, but I'd appreciate more outside confirmations before making official releases. 6.x-1.x-dev or 5.x-1.x-dev should have it sometime...

Also saw:

<IfModule mod_rewrite.c>
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}.gz -f
  RewriteRule ^(.*)\.js $1.js.gz [L,QSA]
</IfModule>

....... I don't think we want that, because that just rewrites requests from .gz to .js. Since we already have AddEncoding gzip .js above that, it handles the gzip. Anyone have any .htaccess gzipping mastery?

After a bit more investigating, it seems we need the additional .htaccess conditions to switch to just plain .js when the browser doesn't support GZip. Like IE6...

Hmm - I posted a comment a while ago, but it disappeared (d.o. DB trouble?).

Anyway, you are correct - there is a reason why the rewrite rule is used, because it can check content encoding, and also serve the file with the correct extension.

The current patch has an major issue in that it does not check that the browser accepts gzip encoding, and is only saving and serving the file with a .js.gz extension, which would break browsers that don't accept gzip and could confuse some (stupid) browsers that don't trust the MIME type alone. I think it should be changed to save the .js.gz file in addition to the plain .js file (as in my original patch) and the rewrite rule described in the original guide should be added.

Here is what each line of the .htaccess code is for:

// If we are serving a .js.gz file
<Files *.js.gz>
// Set the encoding sent to the browser
  AddEncoding gzip .js
// Force the js MIME type (otherwise the browser sees js)
  ForceType application/x-javascript
</Files>

<IfModule mod_rewrite.c>
// If (a) the browser accepts gzip encoded data
  RewriteCond %{HTTP:Accept-encoding} gzip
// and (b) a .gz version of the requested file exists
  RewriteCond %{REQUEST_FILENAME}.gz -f
// Then rewrite the request to seamlessly serve the gzip file instead
  RewriteRule ^(.*)\.js $1.js.gz [L,QSA]
</IfModule>
// If either of the conditions falls through then plain js is served instead (hence why we need to create that file too!)

[Edit for clarity]

We're already serving the gzip JavaScript file that we just created, so the <IfModule mod_rewrite.c> section isn't needed. In fact, it would be better to rewrite .js.gz/.jsmin.js.gz to just .js if browser doesn't support GZip. Would something like this work?

<IfModule mod_rewrite.c>
   # If (a) the browser doesn't accept gzip encoded data
   RewriteCond %{HTTP:Accept-Encoding}!gzip
   # and (b) a .gz version of the requested file exists
  RewriteCond %{REQUEST_FILENAME}.gz -f
  # Then rewrite the request to seamlessly serve the normal JS file
  RewriteRule ^(.*)\.jsmin.js.gz $1.js [L,QSA]
</IfModule>

The first section definitely is required though....

........... You're saying serve it the other way around? Don't serve the GZip file, but have a rewrite to serve it with the rewrite rule?

I heard that gzip is counter productive, because the client is forced to unzip?

Hopefully this won't be a mandatory feature of the module.

The current approach is broken - browsers (especially devices) that don't support gzip will not get any js. Inverting the rewrite rule still wouldn't work, because we are no longer creating a plain text js file to rewrite to. In addition it is still serving the incorrect file extension.

Generally I think progressive enhancement by serving the plain js file is preferable and switching out the compressed version *if* the browser accepts it, than doing the "special case" by default and trying to fix it if it doesn't work. It is also easier to understand and fixes the file extension issue. Most importantly, it also "fails safe", in that if (for example) the php doesn't allow gzipping, or the .htaccess isn't found/readable then the site will continue serving the regular js file that works for everyone.

subscribing...

The current approach is broken - browsers (especially devices) that don't support gzip will not get any js. Inverting the rewrite rule still wouldn't work, because we are no longer creating a plain text js file to rewrite to.

So you're saying GZip compression isn't going to work at all? Or are you saying take this solution:

1. Create both the GZipped version and regular aggregated version
2. Add the .htaccess file which will rewrite to the .gz file if the browser accepts it

PHP 4.0.4 and up support gzencode...

Yes - that is the solution I am proposing :)

@#17

The correct term is "inflate" (the opposite of "deflate"). It's true that inflating takes time, but typically that's far faster than downloading uncompressed files. A run-of-the-mill office PC can easily inflate at 100mb/s. With Deflate compression is fast and decompression is very fast. It also works very well for text based content.

It is somewhat counter productive if it's compressed on-the-fly, since it will stress the server quite a bit (level 9 would be also pretty crazy then). However, in this case the files are only compressed as needed and the resulting gz-files are cached.

The main motivation behind this optimization is to dampen the effect of traffic spikes. When these happen the per-client-bandwidth is rather low, which means that less transferred bytes per page impression can make a big difference for loading times. The vast majority of users won't have a primed cache after all. Additionally, you save lots of traffic, which is of course also pretty cool. (In my case using gzip for JS and CSS as well shaved off almost 50% of the size of a typical blog post. That's pretty hardcore.)

Google for example also gzips its scripts and css files and they probably know what they are doing.

http://drupal.org/cvs?commit=159920
http://drupal.org/cvs?commit=159919

Just committed the fix to both Drupal 5 and 6 branches. It creates the additional GZip file, but serves the regular .js file, as well as sticks in the .htaccess file. It still says I'm not GZipping it through YSlow though, anyone know what's wrong with it?

Here are my 2 cents:

I just added the following line below line 76 of Javascript Aggregator module:

file_save_data(gzencode($contents,9), $jsfile .'.gz', FILE_EXISTS_REPLACE);

It works like a charm for me. It creates an additional GZipped version of the .js file and with a small .htaccess command it serves that gzipped version of js. I shows as GZipped in my YSlow and site speed has improved a LOT.

@oNyx thanks for the explaination.

This should be part of 5.x-1.x-dev and 6.x-1.x-dev by now. Someone mind testing it before we make a release?

After a little poking around I found the problem. We need to add a "RewriteEngine On" after the line. This is unnecessary when using the core .htaccess, because the rewrite engine is already enabled before we add our stanza.

I added a patch to put this in Drupal 7 (for both CSS and JS) at http://drupal.org/node/101227#comment-1171699

Hi, I'm a little confused on whether or not this is working. I've turned on gzip and also applied the change in #24, yslow is still showing me as uncompressed.

I'm also a little confused on how I should set my apache config. Should I turn on mod deflate for js and css? Or just css or neither?

subscribing

subscribing thanks

.htaccess file isn't working correctly. the gz files are being created, but they are not being served. My guess is the RewriteRule. Replaced the .htaccess file in sites\default\files\js with this

<FilesMatch "\\.js.gz$">
  ForceType text/javascript
  Header set Content-Encoding: gzip
</FilesMatch>
<FilesMatch "\\.js$">
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-Encoding} gzip
  RewriteCond %{REQUEST_FILENAME}.gz -f
  RewriteRule (.*)\.js$ $1\.js.gz [L,QSA]
  ForceType text/javascript
</FilesMatch>

via http://zuble.blogspot.com/2007/02/compressed-js-and-modrewrite.html

Works for me!

Replacing the .htaccess file with that doesn't work here in Firefox 3.

Stuck in a fix based off what you have here. Seems to be working! http://drupal.org/cvs?commit=181386

firefox get's an illegal character error. In other words the gzip headers are not being sent, or respected. Overwritten the local .htaccess file again with http://drupal.org/node/290280#comment-1337244

Just confirmed an error exists with this logic

<Files *.js.gz>
  AddEncoding gzip .js
  ForceType application/x-javascript
</Files>

I am pretty sure the only change needed (from the original patch) is turning the rewrite engine on, as I suggested in #27 - without that it won't do much right at all :)

Changing to text/javascript fixed it for me. YSlow is reporting GZip compression with remarkable file size drop. http://drupal.org/cvs?commit=181746

Thanks guys!

One more problem... on my host GoDaddy this doesn't seem to work

  AddEncoding gzip .js

But this does

  Header set Content-Encoding: gzip

Or you can follow the docs and do it like this

AddEncoding x-gzip .gz

via http://httpd.apache.org/docs/1.3/mod/mod_mime.html#addencoding

The Header command gives an Apache error on my setup. Should we go with x-gzip?

yes, x-gzip is what apache.org recommends

Committed! Release-worthy?

Give it 2 days, if it fully works then i would say yes. Each webhost has it's own apache installation as we just found out, trying to find what works for everyone can be difficult.

K got the latest div.
This

  AddEncoding x-gzip .js

Should be this

  AddEncoding x-gzip .gz

.gz not .js

If you do that, I think it's release-worthy. Right now as .js its doing nothing because Files *.js.gz only selects .gz files. Reason it works for you is because your server is set serve any .gz file with the gzip encoding. You could get rid of that line and it would still work for you because of the way your apache sever is setup.

AddEncoding x-gzip .js.gz would probably work, but I haven't tested that.

Committed!

Just moved some of this stuff to the Drupal 5 branch. This it's good for a release of both branches?

I don't have any D5 sites, so I can't test D5 for you. This latest patch makes gzipped js work, so if the code is very similar (5 vs 6), then yes I would say it's ready. You should include some helpful text about having to del .htaccess in /sites/default/files/js if upgrading. You might want to add in a version comment at the bottom of the file, much like core does with it's root .htaccess file. Then if we need to change the file again, we can roll this out better.

patch to del .htaccess file when gzip checkbox unchecked. Should make upgrade path easier since ftp no longer required.

Great idea. Is the patch missing some code? This only has a #weight change.

lol sorry wrong patch. here's the correct one

Fixed! YAY!

There still is one issue... in some rare cases you can have more then 1 js file being served (google analytics module with block only being visible to anonymous). Right now you don't handle that case. I'm waiting CVS access so I can release my css gzip module; which borrows heavily from this one as you can imagine. While writing it, I came up with the del .htaccess file & the ability to process multiple files. Should I roll out a patch to handle multiple js files?

Or is this an issue of google analytics not being added to drupal the correct way? Since there should only be one js file.

Ah yeah, that's true.... Also need to stick this in the Drupal 5 branch, Ii believe.

I think google analytics adds the js with 'inline', but a js file does get served from sites/default/files/js so this might be possible; but i don't see it being added to $variables['scripts'] when I do a print_r(). gzipping everything else is amazing for page load time so supporting google analytics should be a separate issue. I recommend doing a new official release, writing out the upgrade path in release notes so the old .htaccess file gets nuked.

Committed the htaccess delete to the DRUPAL-5 branch: http://drupal.org/cvs?commit=187538

What's the issue you're experiencing? Not quite sure I understand...

in admin/settings/googleanalytics under advanced, set Cache tracking code file locally to true. that js file isn't aggregated and isn't compressed. It does show up in the sites/default/files/js dir though. Probably should open an issue under google analytics.

google analytics problem solved: #411504: Aggregate the 'Cache tracking code file locally' js file

New edge case on Godaddy's deluxe hosting. If you have multiple domains under one account, mod_rewrite rules goof up when placed in a sub dir. If I migrate the .htaccess rules from default/files/js to the root then it works. very weird. if the .htaccess file is still in default/files/js and the rules are in the base .htaccess then it still messes up as well.

Add this to your readme file

Issues
------
Certain hosts do not like multiple .htaccess files. To get around this issue you 
need to copy the text below into drupal's root .htaccess file, and enable the 
"GZip JavaScript: Do not generate .htaccess file" checkbox on the performance page.
Add this Inside the <IfModule mod_rewrite.c> block, right before </IfModule> (add it to the bottom)

  #JavaScript GZIP
  <Files *.js.gz>
    AddEncoding x-gzip .gz
    ForceType text/javascript
  </Files>
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}.gz -f
  RewriteRule ^(.*)\.js $1.js.gz [L,QSA]
  #END JavaScript GZIP

came across an obscure bug. happens if someone enables everything at once
http://drupal.org/node/414120#comment-1432512

Recommend adding in a check for $is_writable; throwing an error to watchdog or something if not.

another fix http://drupal.org/node/101227#comment-1597328

safari works when file extension is .js (and .css)

Lets integrate to core #101227: Gzip aggregated CSS and JS

How would we do that? Change the name of the gzipped files and add the .htaccess stuff from http://drupal.org/node/101227#comment-1597328 to the one created by Javascript Aggregator?

Suppose the right way is to make somefile.gz.js and somefile.js and modify .htaccess

take a look at commit about file names http://drupal.org/cvs?commit=214128 from #452704: Prevent blocking by firewalls compressed js and css files

Is the staus for this issue correct. 6.x-1.2 release notes says this is already commited but the issue is still open?

This was committed a while ago by Rob Loach. Gzip compression works fine. This issue is still open to finetune this feature for various browsers.