Subscribe to Forum One feed
Extend Your Influence
Updated: 2 hours 21 min ago

Bike Spike: Scaling Cascade Bicycle Club’s Web Application

January 28, 2015 at 2:57pm

As the largest bicycling club in the country with more than 16,000 active members and a substantially larger community across the Puget Sound, Cascade Bicycle Club requires serious performance from its website. For most of the year, Cascade.org serves a modest number of web users as it furthers the organization’s mission of “improving lives through bicycling.”

But a few days each year, Cascade opens registration for its major sponsored rides, which results in a series of massive spikes in traffic. Cascade.org has in the past struggled to keep up with demand during these spikes. During the 2014 registration period for example, site traffic peaked at 1,022 concurrent users and >1,000 transactions processed within an hour. The site stayed up, but the single web server seriously struggled to stay on its feet.

In preparation for this year’s event registrations, we implemented horizontal scaling at the web server level as the next logical step forward in keeping pace with Cascade’s members. What is horizontal scaling, you might ask? Let me explain.

[Ed Note: This post gets very technical, very quickly.]

Overview

We had already set up hosting for the site in the Amazon cloud, so our job was to build out the new architecture there, including new Amazon Machine Images (AMIs) along with an Autoscale Group and Scaling Policies.

Here is a diagram of the architecture we ended up with. I’ll touch on most of these pieces below.

Cascade-scaling2 Web Servers as Cattle, Not Pets

I’m not the biggest fan of this metaphor, but it’s catchy: The fundamental mental shift when moving to automatic scaling is to stop thinking of the servers as named and coddled pets, but rather as identical and ephemeral cogs–a herd of cattle, if you will.

In our case, multiple web server instances are running at a given time, and more may be added or taken away automatically at any given time. We don’t know their IP addresses or hostnames without looking them up (which we can do either via the AWS console, or via AWS CLI — a very handy tool for managing AWS services from the command line).

The load balancer is configured to enable connection draining. When the autoscaling group triggers an instance removal, the load balancer will stop sending new traffic, but will finish serving any requests in progress before the instance is destroyed. This, coupled with sticky sessions, helps alleviate concerns about disrupting transactions in progress.

The AMI for the “cattle” web servers (3) is similar to our old single-server configuration, running Nginx and PHP tuned for Drupal. It’s actually a bit smaller of an instance size than the old server, though — since additional servers are automatically thrown into the application as needed based on load on the existing servers — and has some additional configuration that I’ll discuss below.

As you can see in the diagram, we still have many “pets” too. In addition to the surrounding infrastructure like our code repository (8) and continuous integration (7) servers, at AWS we have a “utility” server (9) used for hosting our development environment and some of our supporting scripts, as well as a single RDS instance (4) and a single EC2 instance used as a Memcache and Solr server (6). We also have an S3 instance for managing our static files (5) — more on that later.

Handling Mail

One potential whammy we caught late in the process was handling mail sent from the application. Since the IP of the given web server instance from which mail is sent will not match the SPF record for the domain (IP addresses authorized to send mail), the mail could be flagged as spam or mail from the domain could be blacklisted.

We were already running Mandrill for Drupal’s transactional mail, so to avoid this problem, we configured our web server AMI to have Postfix route all mail through the Mandrill service. Amazon Simple Email Service could also have been used for this purpose.

Static File Management

With our infrastructure in place, the main change at the application level is the way Drupal interacts with the file system. With multiple web servers, we can no longer read and write from the local file system for managing static files like images and other assets uploaded by site editors. A content delivery network or networked file system share lets us offload static files from the local file system to a centralized resource.

In our case, we used Drupal’s S3 File System module to manage our static files in an Amazon S3 bucket. S3FS adds a new “Amazon Simple Storage Service” file system option and stream wrapper. Core and contributed modules, as well as file fields, are configured to use this file system. The AWS CLI provided an easy way to initially transfer static files to the S3 bucket, and iteratively synch new files to the bucket as we tested and proceeded towards launch of the new system.

In addition to static files, special care has to be taken with aggregated CSS and Javascript files. Drupal’s core aggregation can’t be used, as it will write the aggregated files to the local file system. Options (which we’re still investigating) include a combination of contributed modules (Advanced CSS/JS Aggregation + CDN seems like it might do the trick), or Grunt tasks to do the aggregation outside of Drupal during application build (as described in Justin Slattery’s excellent write-up).

In the case of Cascade, we also had to deal with complications from CiviCRM, which stubbornly wants to write to the local file system. Thankfully, these are primarily cache files that Civi doesn’t mind duplicating across webservers.

Drush & Cron

We want a stable, centralized host from which to run cron jobs (which we obviously don’t want to execute on each server) and Drush commands, so one of our “pets” is a small EC2 instance that we maintain for this purpose, along with a few other administrative tasks.

Drush commands can be run against the application from anywhere via Drush aliases, which requires knowing the hostname of one of the running server instances. This can be achieved most easily by using AWS CLI. Something like the bash command below will return the running instances (where ‘webpool’ is an arbitrary tag assigned to our autoscaling group):

[cascade@dev ~]$aws ec2 describe-instances --filters "Name=tag-key, Values=webpool" |grep ^INSTANCE |awk '{print $14}'|grep 'compute.amazonaws.com'

We wrote a simple bash script, update-alias.sh, to update the ‘remote-host’ value in our Drush alias file with the hostname of the last running server instance.

Our cron jobs execute update-alias.sh, and then the application (both Drupal and CiviCRM) cron jobs.

Deployment and Scaling Workflows

Our webserver AMI includes a script, bootstraph.sh, that either builds the application from scratch — cloning the code repository, creating placeholder directories, symlinking to environment-specific settings files — or updates the application if it already exists — updating the code repository and doing some cleanup.

A separate script, deploy-to-autoscale.sh, collects all of the running instances similar to update-alias.sh as described above, and executes bootstrap.sh on each instance.

With those two utilities, our continuous integration/deployment process is straightforward. When code changes are pushed to our Git repository, we trigger a job on our Jenkins server that essentially just executes deploy-to-autoscale.sh. We run update-alias.sh to update our Drush alias, clear the application cache via Drush, tag our repository with the Jenkins build ID, and we’re done.

For the autoscaling itself, our current policy is to spin up two new server instances when CPU utilization across the pool of instances reaches 75% for 90 seconds or more. New server instances simply run bootstrap.sh to provision the application before they’re added to the webserver pool.

There’s a 300-second grace time between additional autoscale operations to prevent a stampede of new cattle. Machines are destroyed when CPU usage falls beneath 20% across the pool. They’re removed one at a time for a more gradual decrease in capacity than the swift ramp-up that fits the profile of traffic.

More Butts on Bikes

With this new architecture, we’ve taken a huge step toward one of Cascade’s overarching goals: getting “more butts on bikes”! We’re still tuning and tweaking a bit, but the application has handled this year’s registration period flawlessly so far, and Cascade is confident in its ability to handle the expected — and unexpected — traffic spikes in the future.

Our performant web application for Cascade Bicycle Club means an easier registration process, leaving them to focus on what really matters: improving lives through bicycling.

Categories: Planet Drupal

Forum One to Host Global Sprint Weekend in DC

January 8, 2015 at 10:49pm

Drupal at Forum One - Powering Problem SolversWant to join me for a marathon next weekend?

As part of a worldwide effort known as the Drupal Global Sprint Weekend, hundreds of coders from around the world are joining together in a united effort to complete a marathon task: the launch of the next generation of the world’s most popular open-source platform, Drupal 8.

To participate in this massive movement and contribute to the Drupal Community, Forum One is hosting a local Code Sprint in downtown Washington, DC on Saturday, January 17th. Sign up here »

Never been to a code sprint before? No worries; we’re pros at these events! Here’s what you can expect:

What is a code sprint?

A code sprint is when developers get together and write code. There’s minor instruction and some ad hoc mentoring, but mainly the focus is just uniting developers and hammering out code together. That’s all there is to it!

How will this event work?

Our developers will work with you to find Drupal 8 core issues for you to focus on. You won’t need to research anything on your own, but you will need to bring your own laptop, and it helps a lot if you set up your development environment beforehand. For instructions on how to get set up and for additional details like the event agenda, visit the RSVP page »

Why should I attend?
  • You’ll meet other DC area developers!
  • You’ll roll up your sleeves and become a bona fide Drupal 8 Core Contributor!
  • You’ll learn from our on-site mentors and fellow developers!
  • You’ll earn tons of karma by furthering the mission of the open-source development model!
  • It’s totally free!

Intrigued? Excited? Can’t hardly contain your enthusiasm? Awesome! Sign up for this free event and join me and the DC open-source developer community as we take on this marathon effort and get 26.2-ish miles closer to bringing Drupal 8 across the finish line.

Categories: Planet Drupal

Using Panels without Panelizer

November 25, 2014 at 10:23pm

The Panels and Panelizer modules have opened up a whole world of options for layouts in Drupal, but too often the usage and features of these powerful tools get confused. My goal with this post is to explore some of the capabilities the Panels module has to offer before ever getting into the realm of using Panelizer.

At a high level, the goals of each of these modules break down into the following:

  • Panels: Create custom pages with configurable layouts and components.
  • Panelizer: Configure Panels layouts and content on a per instance basis for various entities.

Often, I see both modules added and used when Panelizer isn’t needed at all. What’s the problem with that? Introducing Panelizer when it isn’t needed complicates the final solution and can lead to unnecessary headaches later in configuration management, content maintenance, and system complexity.

Panels and Panel Variants

Before the introduction of Panelizer, site-builders got by just fine with Panels alone. This original solution is still valid and just as flexible as it ever was. The secret in doing this lies in understanding how variants work and knowing how to configure them.

Default Layout for All Nodes

Once Drupal finds a Panels page in charge of rendering the page request, Panels proceeds through the page’s variants checking the defined selection rules on each. Starting from the first variant, Panels evaluates each set of selection rules until one passes. As soon as a variant’s selection rules pass successfully, that variant is used and the rest below it are ignored. This is why it’s important to pay attention to the order in which you define your variants to ensure you place less strict selection rules later in the sequence.

Using this to your advantage, a good common practice is to define a default variant for your Panels page to ensure there is a baseline that all requests can use. To do this, you’ll need to define a new variant with no selection rules, so the tests always pass, and place it last in the series of variants. Since the selection rules on this variant will always pass, be aware that any variants placed below it will never be evaluated or used.

Screenshot of variant rule

Custom Layouts per Content Type

Once you have a generic default in place to handle the majority of content items for your Panels page, you can start to tackle the pages that might have more specific or unique requirements. You can do this by creating a new variant above your default and define selection rules to limit its use to only the scenarios you’re targeting.

Screenshot of Panel Content

A common use case for this is the changing layout for content based on content types. To build this out, you need to edit the default node_view Panels page and add a new variant. If this page variant is intended to handle all nodes of this type, I’ll typically name it with the name of the content type so it’s clear. The next step is to configure the selection rules by adding the “Node: Bundle” rule and select the content type we’re building for. Once you save the new variant, any detail pages for that content type should render using the new configuration.

Screenshot of Reorder Variants

Building on this, a single page can be expanded to handle any number of variants using any combination of selection rules necessary. It’s common to see a variant added for each content type in this way. Examples of further customizations that are possible include:

•    A specific layout for a section of the site matching a specific URL pattern

•    Separate layouts based on the value of a field on the entity

•    Alternate views of a page if the user doesn’t have access to it

•    Separate views of a page based on the user’s role

Thanks to CTools, these selection rules are also pluggable. This means if you can’t find the right combination of selection rules to enforce the limitation you need, it’s easy to write a new plug-in to add your specific rule.

Avoiding Too Many Variants

Using the existing selection rules allows for a great deal of flexibility. Adding in custom plugins further improves your options to define any number of increasingly specific variants.

It is possible to take this too far, however. The cost of creating a new variant is that your layouts and content configurations are now forked further. Any common changes across them now have to be maintained in independent variants to maintain continuity.

Visibility Rules

It’s important to also remember the other features Panels offers, including visibility rules. Visibility rules are configured on a per-pane basis inside a specific variant’s layouts. These rules offer the same conditions available in variant-level selection rules. Since they’re configured on individual panes, however, you can focus on the component-level differences between pages. A common use case for this is to use the same node variant for multiple content types with similar layouts and configure the unique panes with visibility rules to limit which pages they show on.

Screenshot of Visibility Rule

To elaborate on this, here’s an example. Assuming a default node variant with a two-column layout, we can define the common elements that all nodes will have, such as the page title, rendered content, and maybe a sidebar menu. If we then add the requirement that all article nodes include a list of similar articles in the sidebar, we can accomodate this by placing the correct pane in the sidebar and adding the visibility rule “Node: Bundle”. We’ll then configure it to use the current node being viewed and limit it to show only when that node is in the “Article” bundle. Now, whenever a node is displayed, it will show the common panes, but the block for similar articles will only show in the sidebar if we’re viewing a article node.

Bundle

Screenshot of Article Bundle Choosing the Right Approach

Once you get to the level of creating panel variants or visibility rules just for single pages, it’s usually time to ask if you’re using the right tool. When you’ve gotten to the point where individual page instances need to be different, you’ve come to the impasse of determining the best approach.

If it’s only a handful of pages that are each unique, then the most straightforward solution may be to create independent Panels pages for each of these.

If instead, individual instances of the same type need different layouts or configurations, then it may be time to install Panelizer to allow instance-specific overrides.

 

Categories: Planet Drupal

The Drupal 8 Decision

November 18, 2014 at 12:20am

From time to time, we all face big life choices. Should I attend this college? Should I take this job? Should I marry this person?

Drupal 8 LogoYet few life choices loom larger for you and your organization in the next year than “When should I upgrade to Drupal 8?”

Well…perhaps the Drupal 8 decision doesn’t quite rank with the others, but for mission-driven organizations, the decision to adopt this major new release is a significant one, with implications for your digital communications for years to come. For most organizations, the upgrade represents a substantial investment that must be planned, scheduled, and budgeted.

In this article, I’ll provide a rapid overview of the promise and challenge of Drupal 8. Then, I’ll lay out the choices you are facing as a current user, or potential Drupal adopter.

The Promise of Drupal 8

Drupal 8, the first major new release in four years, represents a substantial technological departure from previous versions. (More on what constitutes a major Drupal upgrade.)

For marketers and communications professionals, there’s a lot to like. There are over 200 new features and improvements, including a mobile-first approach, in-place editing, and improved accessibility.

For technologists, Drupal 8 offers improved development techniques, as well as including improved APIs and built-in Web services. But D8 also comes with a learning curve. It has an entirely new architecture and methodology. It will take time for your developers to become comfortable with the new Symfony2 components. Drupal 8’s Object-Oriented Programming approach brings increased flexibility for those with the hardcore computer science skills necessary to exploit it.

This infographic (PDF) from the Drupal Association summarizes Drupal 8’s key features.

Drupal infographic

The Challenge of Drupal 8

One challenge for Drupal users is that the release timeline is still unknown. Certainly, we’re getting close. The beta version was released in October 2014, and it’s anticipated that it will be released sometime in 2015.

Historically, migrations from one major version to another are straightforward, but not always non-trivial. Every Drupal website is a conglomeration of the “core” Drupal software as well as typically dozens of add-on “modules” that extend or improve the software’s functionality. For example, the module that runs your fancy homepage carousel in one version may not be updated or tuned for the next version. And while Drupal 8’s upgrade process is improved, timelines and costs for Drupal upgrade projects are as varied as the websites themselves.

It’s important to realize that once Drupal 8 is released, support for previous versions will flag. If you are currently on Drupal 7, with no immediate plans for a major redesign, the issue is less pressing. But for those on earlier versions, you must be planning and budgeting for an upgrade now.

The challenge for digital communication planners depends on your existing situation. Let’s look at the possible approaches — one situation at a time.

Your Website is on Drupal 5

If you are using Drupal 5, your plan is simple. You must upgrade to Drupal 6 as soon as you possibly can.

Drupal 5 was released in 2007 and superseded by Drupal 6 a year later. If your website is running Drupal 5, it hasn’t received any security patches in four years, and is likely already compromised. It’s probably serving as jumping off point for spamming and other nefarious activities. Improving a Drupal 5 site now is difficult, and few reputable consultancies would agree to improve a Drupal 5 site without first upgrading it.

Once you are on Drupal 6, you then must consider upgrading to Drupal 7 or 8 within the next year, as described in the sections that follow.

Your Website is on Drupal 6

If your website is on Drupal 6, you need to plan for an upgrade to Drupal 7 or 8 within the next year. Once Drupal 8 is released in 2015, official support for Drupal 6 core and modules will cease within three months. This means that as security vulnerabilities are discovered, hackers are highly likely to compromise your website for their evil ends, and you will be powerless to plug the holes.

This means that Drupal 6 sites should be planning and budgeting NOW to upgrade to at least Drupal 7 in 2015. You want to be ready to move quickly once Drupal 8 is released. Three months of support is not a long time.

Your Website is on Drupal 7 (Or you are considering Drupal for Your Next Project)

If your site is currently on Drupal you can breathe easier. Drupal 7 has been out for four years, and nearly a million sites are running Drupal 7 core — far more than all previous versions combined.

Drupal core Project usage

It will soon be time for these sites to transition to Drupal 8, but the community will continue to support D7 until Drupal 9 is released, which is surely at least two years away.

Therefore, if you are happy with your existing website and are planning only minor improvements to the design or functionality in the near term, you can sit back and  do nothing for now. You should, of course, start planning for upgrading to Drupal 8 the next two years.

If you are currently considering building a new site on Drupal — or contemplating a major redesign in the next six months — the decision is more complicated.  You have two options.

The first option is to redesign on Drupal 7 now. That’s what thousands of projects are doing as we speak. At Forum One, every new major Drupal project currently starts with Drupal 7, and will likely continue to do so for several months following Drupal 8’s release. The software is mature, stable, widely-supported, and well understood by our staff.

The second option is to postpone your Drupal project until after Drupal 8 is released. Early adopters of D8 will get the maximum value from the new software, as their finished solution will live on Drupal for the longest period. They likely won’t need to consider a new Drupal upgrade until sometime in 2017 at the earliest. And given that the community will continue to support Drupal 8 even once Drupal 9 is released, you would be able to sleep easy knowing that your site will be able to stay patched and secure for the next four to five years.

However, at this writing, there are distinct trade-offs with waiting for D8. You are tying your project timeline to the D8 release schedule, which is community-driven and not guaranteed. Even once D8 is released, it will be a few months before skilled developers are ready to start new projects on 8. While savvy technologists are already experimenting with the D8 beta, it will take some time for modules, processes, training materials, and hosting environments to become tuned for this substantially-new platform.

Here’s another important consideration: Are you the type of product owner who can afford to be on the cutting edge? Early technology adopters typically pay more for the technology than those who follow. Later adopters benefit from the lessons of early adopters. Still, this may be an acceptable premium for your organization if Drupal 8 improves your efficiency, reduces long-term costs, or gives you a competitive advantage in achieving your goals.

Decide to Plan

Like all software, the lifespan of every major Drupal version is limited. Drupal 5 is end of life, Drupal 6 is very near end of life, and — after a good run — Drupal 7 will enter its golden years in the next twelve months.

While the Drupal 8 decision may not have the gravity of other life choices, you have an obligation to ensure that the core software for your site is, secure, stable, and well-supported.

Your most important decision is to decide to plan. Drupal 8 is coming. Are you prepared?

 

Categories: Planet Drupal