Crooks and Liars is an American liberal blog, which was founded in August of 2004, during the 2004 Presidential election, by John Amato. It was the first video style blog around, starting in a pre-YouTube era. Crooks and Liars has a team of about a dozen volunteers, including administrators, contributors and moderators.   Crooks and Liars has grown immensely since its birth, now averaging  over 230,000 unique visitors per day and over 330,000 page impressions.

Evolution

Crooks and Liars originally started out on Radio Userland , which served as its home for two years. After that we started exploring other blogging platforms. At that time we were averaging around 100,000 hits per day. We decided to move to Wordpress , which could handle our smaller team of only 4 at the time.

As the site continued to grow and we were approaching the 200,000 hits per day mark, we started experiencing a lot of down time from server overloads. We were utilizing the famous wp-cache plugin for Wordpress, as well as hosting the database on a single master and two slaves, using the HyperDB class for Wordpress to handle the replication.

While investigating the problems we realized one of our problems was the high level of comments the site receives per day – around 2,000. Being a political blog we also attract a lot of disruptions in our comments, which meant a larger work load for our small moderation team of volunteers.

CMS Selection

Design Evolution

With our increased down time, we had to start looking for another CMS solution. We also had plans on expanding the site beyond just being a political blog, so that meant we need something that could handle our future plans.

I had been running my personal blog on Drupal since the 4.7 days, having moved from Wordpress myself, so I was already aware of a noticeable performance increase plus a greater framework in which to expand upon. This instantly put Drupal ahead of the rest. I then decided to confirm through a series of tests. I setup default installations of Wordpress 2.3 and Drupal 5. I only enabled the core caching mechanisms in both setups and populated them with the exact same data and display options. Both systems also used the default themes and features. After running a series of tests through JMeter, I quickly confirmed my beliefs and even exceeded them as I saw Drupal was able to handle about eight times the requests per second as Wordpress, both on the front page and the same single post view with 157 comments.

Hardware

We are running Crooks and Liars off of four servers. One server is strictly to handle our videos, then the rest are divided up as such:

  • WEB – 2.66 GHz quad code Xeon with 8gb of ram running NetBSD 4.
  • MySQL  - 2.4 Ghz Core 2 quad with 4gb of ram, running NetBSD 4.

We also have an additional server operating as a slave for MySQL right now, although we aren’t actually using replication through Drupal. It is acting more or less as a hotbackup. That same server also runs our Memcached server. We also have an Apple XServe G5 which handles all our static content.

Converting Data

Converting data over for my tests was rather simple. I used the Wordpress to Drupal converter offered by Prime357 , which made the work very simple. However moving over our live data had some extra obstacles. We were using a few special plugins in Wordpress to store our video data in the posts, plus to handle embeds easily. Before doing our final conversion I had to write a couple of short scripts to handle the converting of our media data so a new custom Drupal module I was working on could deal with it. I also decided to remove our embed plugin we had used in Wordpress and instead inject the embed code directly into the node->content field for Drupal. Once that was handled, I ran the Prime357 converter and we had the data fully ready for Drupal. The conversion program took approximately an hour to run on our 600+ megabyte database containing over 30,000 posts and over 800,000 comments. The time was really amazing considering that large amount of data.

Design

Our Wordpress template was pretty much the same as the Radio Userland template we had used prior to switching. With our big move we also decided we wanted a sparkling new design. We enlisted the help of Carolyn King , a highly talented web designed I had worked with before. We wanted to come up with a template that would fit with our genre of being a video site and be able to handle a variety of advertising formats. We also decided it was time to move to a fixed width format, which we decided to size for monitors at 1024x768. For the main site, we had come up with a couple of different designs for the blocks. I added in a custom module so that we could designate available block styles in the template.info file and select our block style in the block configuration page.

All of our templates are sub-templates of a main skeleton. This makes adding in new sites extremely simple, mostly just changing around a few lines in the CSS file.

Contributed Modules

Most of the modules that make Crooks and Liars tick are custom, but we are relying upon a few contributor modules.

CacheRouter – We are using this configured for our memcached server. Currently we are using only a single server, but plan to move it to a couple of servers based upon data type in the near future.

BUEditor – We decided to drop the WYSIWYG feature of Wordpress that was offered through TinyMCE and instead go to a code editor only. The posts on Crooks and Liars do not contain much formatting and we constantly suffered problems of people pasting in code from other webpages into TinyMCE and it breaking our front page. BUEditor’s API also offered the features we were looking for to expand it so we could incorporate our new media management system seamlessly.

We are running a few other contributed modules, such as persistent login, IMCE and IMCE Crop . Most of our expansions are actually done through custom modules.

Custom ModulesMedia Browser

  • CL Media – This is the workhorse of Crooks and Liars. The media system stores data for all our videos (currently over 6,000). A custom slug is inserted into the. When clicking the “Insert Media” button on BUEditor, a new window is opened, designed much the same as the IMCE browser window. From here contributors can add or edit videos and then choose to insert the media into the post with a screen cap or without. Media is inserted via a custom slug in the $node->content, which gets rendered out during the node alter phase.
  • AJAXComments – Our readers were used to the luxury of Ajax Comments that I had instituted on our Wordpress site, so I decided to come up with a system for Drupal. This is something I hope to put into a contributed module in the future. Users can post comments, preview comments and load new comments without refreshing the screen. We also needed to use Drupal’s comment paging with our large number of comments, so this added a little obstacle into the design. Finally I decided on reloading the entire comment division when new comments are loaded,  along with the pager output. We also decided to use the threaded comments feature, but realized not all users like that. To overcome and complaints I added the option for users to change their comment display preferences right on the node page. There is a small button where users can select display order and type. When they do this it is saved in their user data so the preferences stick throughout the site.
  • MultiSite – All our subsites are their own content type. To add a new subsite, I simply have to create a small module defining it, and the data gets read through a custom hook from MultiSite. On Drupal_init, the request URL is parsed to determine if we are loading a multisite site or not. On single node views, the same check is done so that a multisite item isn’t displayed on a different sites template. This module is very lightweight and has been working out grat. Moderator – One of the problems we had with Wordpress was not having the ability to add a moderator only role. The only way we could have actual comment moderators was to make them full site admins. I had done custom hacking to Wordpress to give us this permission, but it was turning into a nightmare to keep up with on every update. Drupal gives us the ability to have comment moderators out of the box. The only thing I added was a new view that shows all comments posted on the site in chronological order, including the content. I also added in a simple banning system, so our moderators can ban a user, which prevents them from posting comments.

  • AJAX Comments PostManager – We rely heavily upon the ability to schedule posts to publish at different times. I originally had us using the Scheduler module, but ended up writing my own custom solution. For users with the “publish {node_type} capability”, they have a box that they can either enter the time/date or use a popup calendar to schedule the post’s publish time. The PostManager also adds new permissions for each content type. These include publish and promote. This gives us the ability to set different roles for all our subsites. We can have an admin handle scheduling and publishing of posts on the Video Café site, but that person doesn’t have the ability to change the schedule of our front page.

Core Modifications

We made a few minor core modifications to help handle our high traffic spikes. The first thing we did was add in a mechanism for our static content. All Javascript, CSS and image files are automatically sent to our static server. We run this via an rsync. We also have a mod_rewrite rule running on the static server to redirect traffic back to the main server if a file doesn’t exist. This eliminates any race conditions from occurring waiting on our rsync to run. Since we are using custom paths on all nodes, we decided to go through and also cache drupal_lookup_path. This will only cache in memcached. If we have to disable memcached for some reason then no caching will occur.  This greatly reduced our number of queries, which now stands at approximately 80 per page.

Training the staff

The contributing and editorial staff of Crooks and Liars come from more political and journalistic backgrounds, so ease of use was a big factor. One concern was the removal of a WYSIWYG editor. Since our posts are rather simple in formatting, the switch to a text only editor caused no problems. To facilitate training of other parts of Drupal and our custom modules, I made a series of short training videos. I used Camtasia Studio for these videos and simply demonstrated different functions, such as moderating comments, scheduling posts, assigning users new roles, etc. These proved to be very beneficial and reduced the problems the staff had with such a major platform switch.

Final Thoughts

We have been running on Drupal for over a month now with no major incidents. The transition went smoother than anticipated. We had no problems on the conversion, and only a couple of very minor problems following the migration, which were mostly caused by coding errors in our custom modules. Even at our highest peak times our database and web servers sit almost idle. We recently moved to the registered comments only and have seen times with over 200 registered users online and over 3,000 anonymous visitors. At that time our server loads were minimal.

Schedule Manager There is still a lot of work to be done. We haven’t started rolling out the new features we were planning on yet, but those will be coming in the near future. Our goal was to do an incremental roll out so we could easily hammer out any problems as they popped up. This has worked out extremely well.

With the minor changes between point versions of Drupal, we have also been able to easily patch our core with our modifications. I can easily update the site to a newer point version within about ten minutes. This was a big selling point and something we couldn’t do on Wordpress with the core modifications we had to make there.

John Amato, the founder of the site, is extremely happy with the outcome. We are looking forward to a long term relationship with Drupal and hope to contribute back to the community.

Comments

zilla’s picture

this is a great write-up - and a few quick comments/questions:

theme at 1024? i notice that many fixed widths are still at 900ish - any reason why you think 1024 is more suitable?

benchmarking: now that you've got years of data, can you comment qualitatively or statistically about how much better or worse drupal is in key areas like site speed, comment posting time, administration of content etc...

btw -have you looked at the "comment notify" module and related queue-mail module (for letting anonymous commenters stay engaged? this is a popular wordpress feature that now may go into the core of drupal7 (per dries) via the 'comment notify' module)
........................................................................
i love to waste time: http://twitter.com/passingnotes

Jamie Holly’s picture

I don't have the original data any longer, but I am planning on redoing it once the holidays are over and time frees up a little bit.

While profiling both code bases, it seems that Wordpress actually takes longer on the content rendering. Posting of comments and content takes about the same time in both systems, but both differ pretty significantly in terms of management. The Manage Content screen on Drupal is a little slower than Manage Posts on Wordpress, where Manage Comments and Users on Drupal are significantly faster. With our number of comments, the comment management screen was still rather slow, so I designed a different interface for it with an index designed specifically for that page. The gave us great performance.

We decided on the 1024 after following our Google Analytics. We actually have less then 0.02% users who have monitor resolutions >1024, and that size handled all our ads, plus wider content area very nicely. Actually 1024 is becoming more of a standard now a days, with the growing popularity of larger monitors.

Having a decent mechanism for registered commenters was a key to us moving over to Drupal, mostly to combat an onslaught of trolls that is common with political blogs. I have been considering the comment notify module for registered users so they can easily know when someone replies to them.

---------------------

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

Hawkeyi’s picture

I like the site a lot and a good way to waste time at work. But I absolutely hate the design.

prezaeis’s picture

i agree
the design makes me want to die in my own vomit

David Strauss’s picture

Please make your posts a little more mature in the future.

3lite’s picture

I agree! I can image how much more traffic will come if the site looked better!

NaplesSEO’s picture

Me too, I always see their stuff homepaged at digg and reddit, I love their logo the best

seutje’s picture

nice, this is one of the few case-studies that focuses heavily on the performance bit and one of the few that isn't solvable with 99% contrib modules

kudos

Jamie Holly’s picture

Thanks! Performance was a huge issue for us. Political blogs are monsters all their own since they are so localized. When we had our busy day last month that saw over 500,000 page impressions, we were getting 60,000+ page impressions per hour for about 4 hours straight. 95% of the time we average only 16,000 page impressions per hour, but wanted to make sure we could handle the huge spikes as they come in.

---------------------

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

Matt Mullenweg’s picture

Always sorry to see someone leave WordPress, but you ended up pretty much the other best place I could think of. Features are a great reason to switch, but scaling doesn't need to be. We host some of the largest poltical blogs like all of CNN's which regularly get thousands of comments per day and we do about a billion pageviews a month on WordPress.com, so here are some tips for future people who may come across this post (some which may be useful to the Drupal community as well):

1. Every release of WP gets faster, so upgrading can get you sometimes significant boosts depending on your bottleneck.
2. Use the memcached object cache backend.
3. If memcached is set up, use Batcache instead of wp-cache.
4. If you get a lot of comments, consider using InnoDB as your storage engine instead of MyISAM inside of MySQL.
5. Double-check that your webserver is set up properly for static requests, this is the cause of 90%+ of the problems we see.

With the above and a single $100/month server from LT you can get around 20,000,000 pageviews a day. With shared Batcache and HyperDB (which you already used, nice) it's a lot easier to scale out both the web and database tier independently as needed. We haven't found the upper limit of this strategy yet.

Mtt’s picture

An open source software that combine the Worpdress supreme magic for spreading words and Drupal Kung-FU modularity, flexibility power for handling massive amount of users and content.

I Know the big red man does *not* exists ..

iimitk’s picture

I've come across this website several times & have been impressed with the activeness of its userbase through the average number of comments on articles & blog posts.

A good thing the study didn't mention is that the website runs on the latest & greatest Drupal 6 (Specifically 6.6 according to CHANGELOG.txt :-)). It's always nice to see active & heavy-trafficked websites taking the D6 route.

I've a couple questions & remarks:

1) Is the number 230,000 unique visitors a day correct? It looks odd that these users only view 330,000 pages, which means ~1.4 pageview per visitor, a number contradicting the amount of activity apparent in comments. I think you meant 23,000 visitors, didn't you? If you do have 230,000 unique visitors/day, you really shouldn't be sitting here writing case studies, unless you did so from some Caribbean island for just wasting time. ;-)

2) Do you have any plans for releasing your custom modules as contributions for the community? Since you're saying it's all a voluntarily effort, it's needless to say that you'd benefit more by releasing these modules to the wider Drupal community.

3) I couldn't find any information about the website's purpose, mission, story, team or other related information. The first entry in the FAQ is about "advertising", which isn't the appropriate position, IMO.

4) In the displayed node teasers, the path to every full node is aliased (e.g. mike-finnigan/mikes-blog-roundup-56), while the path in the little dialogue box with the comment count on the right is not aliased, showing Drupal's original internal path (e.g. node/24361). First, this is confusing, and I cannot see why all links wouldn't be aliased. Second, since that box shows the comments posted to nodes so far, wouldn't it be more intuitive to make the link goes to the comments directly? I.e http://crooksandliars.com/node/24361#comments.

5) I'm not a big fan of the design either. Looks a bit clumsy & 1998ish. I think it needs a face lift-up with an emphasis on information architecture & usability. In its current state, it's just a flat blog, and do not make any benefit of some of Drupal's powerful assets like blocks, stats & complex layout theming capabilities.

Keep up the good work.
________________________
"Creativity is knowing how to hide your resources" - Albert Einstein.

Sownn’s picture

the data from sitemeter on that website show that about 11,000 UV per day.

Jamie Holly’s picture

1) I actually wrote this up last month and at that time our SiteMeter was showing an average of 230,000 unique per day. Of course that was reflecting huge spikes in traffic from the election. Most political blogs have now seen a traffic decline of 30-50% following the election (I guess everyone is burnt out). Right now we are at 112,000 per day according to SiteMeter.

2) I am looking at releasing the AjaxComments in the future. This was something I started on Wordpress over two years ago and the users really like, so I think it would be very beneficial as a contributed modules.

3) That's something John Amato, the site owner, is working on. Actually we are going to have an entirely separate "about us" section once it's all done. The Book module works out great for this.

4) Thanks for pointing that out. I actually never noticed it and just now fixed it.

5) Overall, the site is a blog, so that's why we have that design. We are planning on doing a new front page in the future, emphasizing content from all our sites. We got two more sites going live in the next week, so once we have enough content from them and get all the other plans rolled out, then we are going to start looking at a front page design more like one that Huffington Post or Townhall uses - which would display content from all the sites. It was more or less just giving us the time for the other sites to get populated with content, since when we moved over it was only the actual Crooks and Liars blog. Everything else has been added since our switch.

Thanks for the feedback!

---------------------

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

superjacent’s picture

Great write up and thanks for the plug re - the converter, I'm pleased it all worked out.

___________________________

Steven Taylor
http://prime357.net

dman’s picture

Nice write-up, thanks for that.
Good to see a sensible scaling worked for something that size.
Sounds like you made a good use of available modules, and a conservative amount of custom coding. A pretty good balance from the way you describe it!

.dan.

Sree’s picture

Nice writeup .... !

I have a question here - did you faced any junk characters related problems while migrating data from wordpress to drupal?

-- Sree --

-- Sree --
IRC Nick: sreeveturi

Jamie Holly’s picture

None at all. The trick was to convert our WP database over to UTF-8 first, then do the conversion over to Drupal. There's actually plugins available for it here.

---------------------

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

Sree’s picture

Actually I also worked on a project quiet a few months back which also needed the same migration from WP to drupal. So, in this process we faced few problems with junk characters getting inserted in places of special characters. We also updated WP database over to UTF-8 , then over to Drupal but, still needed some more cleaning in DB. So wanted to checkout if you people also faced any such issues ....

-- Sree --

-- Sree --
IRC Nick: sreeveturi

guidot’s picture

As I understand you modified the drupal core code. If so, could you explain why that was unavoidable?

And further, I assume you test your modifications after a core update bevor going live. Does that fit into the ten minutes update procedere you mentioned?

Jamie Holly’s picture

Our core modifications are so simple and minimal that it takes only a couple of seconds to test. For the static URL's, pretty much viewing the site after the modification tells me if it works or not. The caching of path's is really quick and easy to test also. Create a quick node, view it a couple of times with query output enabled in Devel and make sure it's not pulling the path's from the database, then edit the path on my test node and repeat the test to make sure it updated ok.

I originally did these core modifications in 6.2 and haven't had a problem since. I would actually like to see them make it into the core at some point. Path caching can greatly reduce the number of database queries, but it's really only beneficial if you are using a caching mechanism other than the database.

---------------------

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

GSegree’s picture

nice article,

I'm looking to use drupal for our corporate intranet and i'm curious about what training/books or info helped you get up to speed with drupal in building your site?

amanire’s picture

Wow, the timing of this is quite ironic. Earlier this evening, I was reading an article on Technorati Top 100 of blogs broken down by CMS and was disappointed to see so few Drupal entries and the only political blog, NewsBusters, a right wing mockery of Media Matters. I actually quit watching Crooks & Liars after they switched to Wordpress, mostly because I can watch full episodes of Daily Show and Colbert online. I'll have to check C&L more frequently to see the new features. Interesting that you're making the wonks use BUEditor. Reminds me of the day a misplaced teaser break turned my client's entire site bold. Also, I had no idea of Drupal's performance advantages over Wordpress. Fascinating. Thanks!

Jamie Holly’s picture

The improperly positioned break thing plagued us on Wordpress. We would end up with posts below one with a break inside of blockquotes, or all styled. We haven't had any problems since moving to Drupal. I actually have been working on getting TinyMCE for them now, but most of the people would rather stick with the BUEditor - mostly since it loads so much faster.

---------------------

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

chrisroditis’s picture

Nice case study, I once converted a blog from wordpress to drupal 5 and it was a pain. The thing is it was not in english and encodings were messed any way i did it. I am glad there is Prime357 nowadays.

Template Monster templates 20% cheaper
Christopher Skauss:my personal blog

A healthy disregard for the impossible.

Jamie Holly’s picture

I went through that a couple of years ago when I moved my personal blog over from Wordpress 2.0.X to Drupal 4.7.X. The converted from Prime357 is a life saver. I also tested it in Wine and in OS X, running in Parellels, and it worked without flaw on each system.

---------------------

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

vkr11’s picture

Good case study !!

- Victor
Better Way to Search Drupal.org | Lamingo | Income

mroswell’s picture

I've been a daily Crooks and Liars reader for a few years. More-than-daily during election season!
I recommend the site.

taqwa’s picture

2 questions:
1) Which version of Drupal are you using? and
2) When do you think you can have the ajax comments module available for the Drupal community?

phdhiren’s picture

It seems to use the Drupal 6.8 http://crooksandliars.com/CHANGELOG.txt

neochief’s picture

AJAX Comments already at Drupal.org:
http://drupal.org/project/ajax_comments

blavish’s picture

Thanks for sharing.

mishakogan’s picture

I really like your write-up but I think everyone will agree that design is the least attractive thing on the site. I wonder what were your vision for site design and did you do any user-tests before releasing new design?

raz21’s picture

Drupal is more powerful, than Wordpress

------------
estate21

Fred125’s picture

Yes drupal is more powerful

restyler’s picture

Just for note, ajax comments are already there - http://drupal.org/project/ajax_comments

RussianWebStudio: improving the web