Needed for high-performance sites or sites with a lot of assets on the CDN.

Follow-up for #974350: Far Future setting for Origin Pull mode. Partial versions available at #974350-36: Far Future setting for Origin Pull mode and #974350-57: Far Future setting for Origin Pull mode. The former is not very complete, the latter is more complete, but not entirely.

What I've tried to add:
- ability to apply the far future expiration files only to files with certain extensions (and because of this, the rules need to be *generated*
- by default files to expire immediately, only if extensions match far future expiration

What still needs to be added:
- ability to detect If-Modified-Since requests and automatically answer with a HTTP/1.1 304 Not Modified and not perform the actual check; the URL structure already guarantees this
- gzip-encoded content negotiation support
- the code to actually generate this automatically

Files: 
CommentFileSizeAuthor
#16 cdn-1413156-16.patch5.4 KBDmitriy.trt
PASSED: [[SimpleTest]]: [MySQL] 57 pass(es).
[ View ]
#11 cdn-1413156-11.patch4.59 KBDmitriy.trt
PASSED: [[SimpleTest]]: [MySQL] 44 pass(es).
[ View ]
far_future_htaccess.patch4.41 KBWim Leers
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch far_future_htaccess.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Comments

mikeytown2’s picture

304 in htaccess doesn't look good
http://stackoverflow.com/questions/8595970/is-it-possible-to-return-a-30...
http://web.archiveorange.com/archive/v/waCg9GS8iDQRRTyqHvKn

Is it faster to send the whole file, or to boot up PHP to send a 304? I don't see a way to have apache send out a 304; PHP has to be the one to do it. And no, apache does not support reading the file size out of the box http://webkist.wordpress.com/2008/02/01/apache-mod_rewrite-filesize-comp.... I would lean towards sending the file again being faster. Making this an option and letting the admin pick "PHP 304" or "send file again" is probably the best way to move forward though.

For those wondering, these are the 2 ways I've tested some things out

# Test 1.
RewriteCond %{HTTP:If-Modified-Since} .*
RewriteRule .* - [R=304]

# Test 2.
SetEnvIf If-Modified-Since .* CDN_IMS
header set HTTP/1.1 "304 Not Modified" env=CDN_IMS

PHP for a 304; put in hook_boot most likely.

<?php
if (isset($_SERVER['HTTP_IF_MODIFIED_SINCE']) && strpos($_SERVER['QUERY_STRING'], 'cdn/farfuture')) {
 
header($_SERVER['SERVER_PROTOCOL'] . ' 304 Not Modified');
 
header('Vary: If-Modified-Since');
  exit;
}
?>
Wim Leers’s picture

Sending the file again is by definition not faster. More bytes to send. It's only equally fast if the TCP window size is >= the file size + headers.

Note that the above articles don't apply to us as far as I can tell: we would only send 304's for URLs who are by definition unique, so it's okay to blindly send 304's.

Or what am I missing?

mikeytown2’s picture

There is no way to blindly send a 304 in apache; that requires PHP.

Wim Leers’s picture

Aha! Gotcha now :) It's probably faster to just continue to rely on Apache for the 304's then? Even if it's not blindly, it's probably faster than executing some PHP code.

Wim Leers’s picture

acouch’s picture

I had the same problem as #1417616: CDN + Far Future expiration in some cases: PHP allocates all available CPU and memory As soon as I turn on far future expiration the server falls over.

Turning on those rewrite rules did not help much. It seems like apache still can't handle serving the far future url file, I guess because there are too many and it starts slowing down for all of them. It is hard to troubleshoot because as soon as I turn it on the server runs out of memory and I get a number of "Error Request couldn't be satisfied" from Cloudfront.

FYI, in the D6 version:

RewriteCond %{REQUEST_URI} ^/cdn/farfuture/([0-9a-zA-Z])*/(.+)$
woud be
RewriteCond %{REQUEST_URI} ^/cdn/farfuture/mtime\:([0-9a-zA-Z])*/(.+)$

for those trying this.

I'll try and report back with any more findings. Without future expiration things work great.

Wim Leers’s picture

We should also do Header set Access-Control-Allow-Origin * for #982188: Work around Same-Origin Policy: Cross-Origin Resource Sharing (CORS) support.

Wim Leers’s picture

The .htaccess file of HTML5 Boilerplate may be helpful here: https://github.com/h5bp/html5-boilerplate/blob/master/.htaccess.

JamesOakley’s picture

Hmm: This helped me with #1790348: Far-Future mode: background images and font files referenced in CSS files incorrectly rewritten, but only in an Omega subtheme.

I only needed the alteration to my .htaccess file in order to get certain files parsed that were otherwise failing. For the purpose I had, I actually needed to change the line referred to in #6 to

RewriteCond %{REQUEST_URI} ^/cdn/farfuture/([0-9a-zA-Z_])*/mtime\:([0-9])*/(.+)$
RewriteRule .* /%3 [L,E=FARFUTURE_CDN:1]

(There's an alphanumeric string after farfuture and before mtime, and that also includes an underscore character. There's then a numeric string after the mtime reference.)

Wim Leers’s picture

Component:Origin Pull mode» Origin Pull mode — Far Future expiration
Dmitriy.trt’s picture

StatusFileSize
new4.59 KB
PASSED: [[SimpleTest]]: [MySQL] 44 pass(es).
[ View ]

Code from original issue failed for gzipped CSS/JS in our case because of multiple REDIRECT_ prefixes generated by our complex htaccess configuration and missing extensions with .gz suffixes in suggested configuration. Here is a patch with updated version we use. Instead of duplicating all rules it restores original environment variable from the one with REDIRECT_ prefix and it works on large number of internal redirects.

Also, regexp is simplified a bit. Far future headers are added for two additional extensions: .css.gz and .js.gz. Added a note saying that configuration should be added to the <IfModule mod_rewrite.c> section of htaccess file.

Wim Leers’s picture

OMG! AWESOME, Dmitriy.trt, thank you!

Also, regexp is simplified a bit. Far future headers are added for two additional extensions: .css.gz and .js.gz.

Does that solve this:

- gzip-encoded content negotiation support

?

- ability to detect If-Modified-Since requests and automatically answer with a HTTP/1.1 304 Not Modified and not perform the actual check; the URL structure already guarantees this

This has not yet been addressed right?

I'm very tempted to commit this as-is, but I'm afraid people will expect it to cover the full feature set, which it does not.

Dmitriy.trt’s picture

It just adds support for gzipped CSS/JS files generated by core on aggregation (CSS is actually generated by CDN module override). Support for other kinds of gzipped files that should be generated on the first request is still missing.

Support for Access-Control-Allow-Origin header and If-Modified-Since detection is missing too.

Dmitriy.trt’s picture

Security token is useless if we use htaccess, it only makes URLs to be much longer. And URLs are still safe without it because files are delivered by web-server and private filesystem is protected by another htaccess inside its root.

Wim Leers’s picture

#13: Hurray for CSS/JS! It doesn't do *content negotiation* though AFAICT. Regarding Access-Control-Allow-Origin and If-Modified-Since: could you add those? :)

#14: True. But I prefer to keep it anyway, so that you can easily switch from PHP-served to Apache-served, without having to mess around with configuration/caches/etc.

Dmitriy.trt’s picture

StatusFileSize
new5.4 KB
PASSED: [[SimpleTest]]: [MySQL] 57 pass(es).
[ View ]

Do you mean serving gzip-encoded content depending on client support by "content negotiation"? For aggregated CSS/JS it is handled by original Drupal 7 htaccess rules and duplicating those rules is not what we want.

Attaching updated patch with Access-Control-Allow-Origin header and If-Modified-Since detection.

BTW, curl is useful for debugging If-Modified-Since header. Here is the command I've used:

curl -v -H "If-Modified-Since: Wed, 20 Jan 1988 04:20:42 GMT" -o /dev/null http://YOUR.HOST/cdn/farfuture/0/mtime:12345/sites/default/files/PATH/TO/FILE
Wim Leers’s picture

Status:Active» Reviewed & tested by the community

This is AWESOME stuff! Thank you!

I will commit it to version 2.7, together with #1060358: CDN and SEO. I'll mark it as an "experimental feature", so that we can gather more feedback. Unless you're confident that it's rock solid already? :)

Dmitriy.trt’s picture

"Experimental" is the right status for this feature, some feedback about using these rules together with some complicated htaccess files could expose issues we don't know about.

Thanks for your work on this very useful module!

I'm not really sure that Accept-Ranges header is necessary here. It should be returned by the webserver itself if support is enabled and we shouldn't force it.

Wim Leers’s picture

Indeed :)

omega8cc’s picture

Here is Nginx version (needs testing):

###
### CDN Far Future expiration support for Nginx.
###
location ^~ /cdn/farfuture/ {
  access_log off;
  log_not_found off;
  gzip_http_version 1.0; ### if you have: gzip on;
  if_modified_since exact; ### this needs testing - other options: off | before
  add_header Access-Control-Allow-Origin *;
  add_header ETag ""; ### For Nginx < 1.3.3
  ### etag off; ### For Nginx >= 1.3.3
  add_header Vary "Accept-Encoding";
  add_header Accept-Ranges "bytes";
  add_header X-Header "CDN Generator 1.0";
  location ~* ^/cdn/farfuture/.+\.(?:css|js|jpe?g|gif|png|ico|bmp|svg|swf|pdf|docx?|xlsx?|pptx?|tiff?|txt|rtf|class|otf|ttf|woff|eot|less)$ {
    expires    max;
    add_header Cache-Control "max-age=290304000, no-transform, public";
    add_header Last-Modified "Wed, 20 Jan 1988 04:20:42 GMT";
    try_files  $uri @drupal;
  }
  location ~* ^/cdn/farfuture/ {
    expires    epoch;
    rewrite    ^/cdn/farfuture/[^/]+/[^/]+/(.+)$ /$1 last;
    add_header Cache-Control "private, must-revalidate, proxy-revalidate";
    try_files  $uri @drupal;
  }
  try_files $uri @drupal;
}
brephraim’s picture

Anyone test the nginx?

omega8cc’s picture

Yes, we are using a slightly modified and already tested version in the BOA project (and also on our own website): http://drupalcode.org/project/octopus.git/blob/HEAD:/aegir/conf/nginx_oc...

Note that it assumes that you are using Nginx >= 1.3.3 - otherwise you would need to disable etag as explained in #20 above.

It is RTBC already.

omega8cc’s picture

Tested and working version:

###
### CDN Far Future expiration support.
###
location ^~ /cdn/farfuture/ {
  tcp_nodelay   off;
  access_log    off;
  log_not_found off;
  etag          off;
  gzip_http_version 1.0;
  if_modified_since exact;
  location ~* ^/cdn/farfuture/.+\.(?:css|js|jpe?g|gif|png|ico|bmp|svg|swf|pdf|docx?|xlsx?|pptx?|tiff?|txt|rtf|class|otf|ttf|woff|eot|less)$ {
    expires max;
    add_header X-Header "CDN Far Future Generator 1.0";
    add_header Cache-Control "no-transform, public";
    add_header Last-Modified "Wed, 20 Jan 1988 04:20:42 GMT";
    rewrite ^/cdn/farfuture/[^/]+/[^/]+/(.+)$ /$1 break;
    try_files $uri @nobots;
  }
  location ~* ^/cdn/farfuture/ {
    expires epoch;
    add_header X-Header "CDN Far Future Generator 1.1";
    add_header Cache-Control "private, must-revalidate, proxy-revalidate";
    rewrite ^/cdn/farfuture/[^/]+/[^/]+/(.+)$ /$1 break;
    try_files $uri @nobots;
  }
  try_files $uri @nobots;
}

For config context see: http://drupalcode.org/project/octopus.git/blob/HEAD:/aegir/conf/nginx_oc...

NOTE! I have removed above directives which are not related to the CDN support and are used only in the BOA stack context.

Fabianx’s picture

Would be nice to commit the RTBC patch!

It works great!

mikeytown2’s picture

In AdvAgg I did encounter an issue with setting the Last-Modified header to be in the past.
#1431668-4: can't get browser control cache with bundle : always OK 200 reponse

This isn't an issue with the PHP route as inside of cdn_basic_farfuture_download() it will always send a 304. With Apache this can't be done (#3) and thus faking the Last-Modified header prevents a 304 from being returned.

Wim Leers’s picture

Title:.htaccess rules generator for Far Future expiration» .htaccess rules for Far Future expiration: make it possible to use the Far Future feature directly in Apache, avoiding PHP
Issue summary:View changes
Status:Reviewed & tested by the community» Fixed
+++ b/README.txt
@@ -214,6 +214,107 @@ If you just use the CDN's URL (e.g. myaccount.cdn.com), all cookie issues are
+Apache users: add the following rules to <IfModule mod_rewrite.c> section of ¶
...
+    # The redirect works only if URL was actually modified by rewrite rule ¶

Fixed trailing spaces on commit.

  • Wim Leers committed 360ca19 on 7.x-2.x authored by Dmitriy.trt
    Issue #1413156 by Dmitriy.trt, Wim Leers: .htaccess rules for Far Future...
Fabianx’s picture

Nice! Makes me very happy! :) Been using that .htaccess far-future since a while already :).

Wim Leers’s picture

Awesome :)

Status:Fixed» Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.