ESI: Edge Side Includes

Introduction

ESI - or Edge Side Includes - is a high performance caching solution for Authenticated users but can be helpful for Anonymous users as well.

Typically, pages which are personalised for authenticated users (even minor personalisations, such as a block which says "Logged in as manarth") will prevent reverse-proxies (which can easily perform 100 times faster than Drupal) from caching the page, because messages intended for one user could then be seen by another.

ESI is a W3C standard to address this issue. The personalised content is replaced with an ESI tag - <esi:include src="http://example.com/esi/block/xxxxxx" /> - which is identical for every user; the page can then be cached. Edge devices such as Varnish fetch the ESI URL, and merge its contents with the cached page (and the contents of that ESI URL may also be cached).

This technique can easily improve performance by 10x, 100x, maybe even more.

Tag support

The ESI module supports three types of server/edge includes tags:

• ESI: standard Edge-Side Include tags
  <esi:include src="http://example.com/esi/foo" />
• SSI: Server-Side Include tags (typically used on nginx cache servers)
  <!--# include virtual="http://example.com/esi/foo" -->
• Remote SSI: An alternative SSI tag syntax, used by mod_publisher
  <!--# include url="http://example.com/esi/foo" -->

ESI-capable edge devices

• Varnish is a mainstream reverse proxy with native ESI support and significant Drupal integration. Recommended.
• Apache, using mod_publisher to support Remote SSI tags.
• The Nginx web server is supported by this module (using SSI tags instead of ESI tags).
• Squid cache has native support for ESI from Squid 3.0.

ESI-capable services

• Akamai: a global CDN with extensive ESI support.
• Fastly: a SaaS CDN based on Varnish (and used by drupal.org).
• section.io: a SaaS CDN that uses Varnish.

If you know of any other proxies or services that provide native ESI support, please let us know!

Resources

• ESI, Varnish and Drupal slides from a Drupalcamp Toulouse presentation (2011).

Can you help?

The ESI module has a few co-maintainers, but none of us are paid to work on ESI full-time, so end up fitting in bug-fixes and changes around our paid projects (and occasionally corporate sponsors pay for development of specific features, which we're always grateful for).
So we'd love some help to make the module even better!

Specifically, our biggest challenges right now are:

• Tests. There's a lot of complex code, and very little in the way of tests for the system. In particular, panels integration could use a lot more test coverage (and I'm sure there are bugs in there that we simply haven't found, because it lacks test coverage).
• D8. We haven't begun work on a Drupal 8 port yet. Also, we don't know whether it'd be a fairly direct port, or if the new HTTP kernel gives us a better place to integrate, which would imply a big architectural shift. Right now, the most useful task would be an investigation into the D8 HTTP kernel, to understand whether that's a promising integration point.

If you'd like to help, you can:

• Create tickets with bugs you've found/comments/suggestions in the issue queue
• Send patches via the issue queue (including documentation improvements)
• Find manarth (timezone: Europe/London) in #drupal-contribute on IRC if you have any questions/comments

Releases / Maintenance / Development status

• 6.x-1.x [Obsolete]
• 6.x-2.x
  Provides ESI integration with blocks, panel panes, and the context module.
  Recommended for Drupal 6, but no significant development is planned. Consider upgrading Drupal to 7.
• 7.x-3.x
  Provides ESI integration with blocks, panel panes, and the context module.
  Recommended for Drupal 7. Actively developed.

Roadmap

7.x-3.x is being actively developed. These are some of the ideas/features planned:

• ESI fast cache - serving ESI fragments from the cache (even for authenticated users).
• Auth page-cache - serving full pages from the cache (where all personalised sections have been replaced by ESI tags).
• Tests - the ESI block module has a test suite; tests for the other modules are planned.
• Varnish vmod integration - Varnish 3 has a memcache vmod available, so varnish could poll memcache for an ESI fragment, which would be quicker than requesting an ESI fragment via a HTTP request and Drupal bootstrap.

Credits

• 6.x-1.x originally developed by manarth.
• Maintained by manarth, dstuart, and mikeytown2.
• Development of 6.x-1.x funded and sponsored by Axis Twelve Ltd and the Department for Education (UK)
• Development of 7.x-3.x funded and sponsored by Code Enigma and French newspaper Le Figaro