Update: See the Drupal Association Webcast: Fastly takes Drupal.org to Extremes

Fastly logoWe are so stoked to announce our partnership with Fastly. Fastly is now serving up all of our traffic from the *.drupal.org domain Drupal sites and related services.

Drupal.org is big

Drupal.org and its services handle over 1.5 billion requests per month; this is a massive amount of traffic for an open source project.

Every time cron runs the update service on your Drupal site, it talks to updates.drupal.org. Every time you download a copy of Drupal or any projects on Drupal.org, you talk to ftp.drupal.org. (We see over 400,000 downloads of Drupal core in a typical month—way more around DrupalCons and major community sprints.)

Each month we have over 15 million unique pageviews on Drupal.org—by over 2 million unique visitors.

All of those stats are about to rise significantly with the launch of Drupal 8. Drupal 7 caused a 30% bump in traffic when it was released and we expect even more with the launch of Drupal 8.

Fastly is… well… fast

In 2014, we implemented a CDN (content delivery network) for Drupal.org. The impact was immediate. Everything was faster. We met our initial goal of getting Drupal.org sites and services fronted by a CDN.

That CDN solution was a good start, and while it lacked features we didn’t know we needed, it improved our ability to deliver Drupal-generated content as well as packaged projects.

Late in 2014, we were introduced to Fastly. They offered us an opportunity to try out the service for ftp.drupal.org and we haven't looked back.

Varnish for the win!

You may already use the open source Varnish for your Drupal sites. Varnish is an HTTP accelerator. Drupal.org began using Varnish in 2009 to reduce load on its web servers. It is a powerful cache that likely sits in front of your web origins. Fastly gives us a globally distributed Varnish cache with all of the features we are familiar with.

Fastly hires maintainers of the Varnish project and are helping move it forward. Drupal 8 cache keys are also a straight correllation to Fastly Surrogate Key purging and Fastly's active involvement is another great example of Fastly working with an open source community (ours!) to build a better experience.

Yay, open source!

Open Source Alliance

Speaking of open source, Fastly has an open source alliance that provides free content delivery to projects like Debian, the MIT Media Lab, the W3C, Memcached, Linux Foundation, and more. They have even open sourced some of their technology stack to make it easier for others to build systems with their tools. We love partnering with these sorts of organizations.

Supporting Technology Partner

Speaking of partnering, Fastly took their partnership a step further and are helping fund a lot of great work on Drupal.org through commitments to sponsor DrupalCons and even the release of Drupal 8. Look for them at upcoming DrupalCons. Thanks Fastly!

How Fastly Works

Fastly’s Global Network consists of Points of Presence (POPs) running Fastly’s custom Varnish software stack. These POPs are spread around the world in strategic locations close to the highest density Internet Exchange Points, ensuring cached content is just a few milliseconds away.

Global network

Fastly has a powerful feature that allows us to specify a specific POP as a shield to our origin server. In our case we chose the Seattle POP because of its proximity to our servers at the OSL in Corvallis, Oregon in the United States.

Caching basics

The origin shield configuration means all requests to Drupal.org flow through Fastly’s origin server in Seattle before reaching our Drupal.org origin. If Fastly’s origin server in Seattle has already cached the file, no request to our Drupal.org origin is necessary.

Caching basics, subsequent requests

In practice, this allows us to deliver a huge amount of content with an extremely high hit ratio. (See that spike in the GIF below, that's one of those regular times that cron jobs request a ton of content from our updates server. Fastly is just churning along.)

Really cool animated GIF showing updates traffic

(Fastly made us confident we could put this enormous GIF in our write up.)

Faster changes, faster response time

Another advantage to Fastly has been the efficiencies it has given us by allowing us to quickly make changes to how they cache our content. With our previous CDN, changes could take up to 4 hours to propagate through the system. We can now change a Varnish config and reflect those changes in under 5 seconds. That has been a huge help when we need to make a hot fix to production and keep Drupal.org delivering Drupal to the world.

Logging and improving our usage stats

Fastly also allows us to stream our logs. This means we can parse those logs and turn them into data. We are already seeing much more reliable project usage statistics with from our new log processing for updates.drupal.org thanks to Fastly’s log streaming.

More accurate logs mean we can provide users with better data to help them to make better decisions about modules, themes and distributions that have the most installations.

Next Steps with Fastly

We have been impressed with Fastly, both for their responsiveness and the performance improvements we have seen while using their services. They have given a ton to our community and we can't wait to see where they take Varnish and their open-source-powered service. If you are interested in using Fastly with your own Drupal sites, you can learn more about Fastly on their Drupal.org organization page or sign up for a developer account at Fastly.com—there is even a Fastly module for Drupal7, and one for Drupal8 is on the way!

Comments

Jo Wouters’s picture

Please note the URL in the text "How Fastly Works: Fastly’s Global Network" includes a double quote, and is not valid ( https://fastly.us/1kv5OAw" )

joshuami’s picture

Thanks for that! All fixed.

gokulnk’s picture

Interesting insights and a well written article. Will start looking into Fastly and see how it fares with other CDNs. Remote log streaming also seems to be useful. Time to check it out.

christefano’s picture

Many thanks to Fastly for being our marquee sponsor for the Drupal 8 Release Party: Los Angeles!

While this event is mostly a networking event and celebration, Fastly will be demonstrating their Drupal module (available for both Drupal 7 & 8) that integrates their real-time CDN with Drupal.

Guest Speaker: Rogier "Doc" Mulhuijzen, Varnish Wizard, Fastly Senior Engineer
Topic: Fastly's Drupal 7 & 8 Plug-ins

Everyone is welcome. Please RSVP so we know to have enough food and drinks: http://drupal8releasepartylosangeles.eventbrite.com

mradcliffe’s picture

Nice. It was fairly seamless and I completely forgot I had fastly in my hosts file.

gopagoninarsing’s picture

Interesting.Time to check it out

Amazon’s picture

Hi, this is great news. You can check out the Fastly and D8 demo from Wim and Fabian's cache presentation at Drupalcon Barcelona.

https://events.drupal.org/barcelona2015/sessions/caching-edge-cdns-everyone The Fastly demo starts around 30:00.

Fastly is also sponsoring the Drupal 8 release party in San Francisco! I hope everyone enjoys going to the party today.

Kieran Lal

rivimey’s picture

The visible text of the ink is correct but the link itself is different, and wrong.

This is correct: https://events.drupal.org/barcelona2015/sessions/caching-edge-cdns-everyone

Zirky’s picture

I am totally new in Drupal but it will help me a lot so, great news !

rivimey’s picture

Not sure if this is the right place, but some embedded-in-texr images are now broken. The affected images all have a src url starting "/files", whereas working images start with "http:" or "https:".

I have particularly noticed this on the association news pages on this link : https://www.drupal.org/aggregator/sources/628

which i reached from the https://www.drupal.org/news page and then clicking on the Drupal Association link in the right sidebar.

drupalnewie @berfi’s picture

Is it possible to share the architecture for database clusters, webservers and all technologies which help this massive system handles that much traffics with such a best performance?

Thanks

basic’s picture

I believe the Drupal.org hardware information is available here: https://www.drupal.org/drupalorg/hardware

It would be nice if we wrote up a blog post with more detail at some point as well.