By bertboerland on

Just a thread to show what sites have been under extreme load from for example slashdot.

For example, on February 4th, 2005 Wikinerds Portal, a Drupal site running an article on GNU/HURD got hit rather hard.

If you know a Drupal site that has been /.-ed or better, if your Drupal site has been /.-ed and you want to share your experience, post it here.

Comments

nsk’s picture

nsk commented

I am the admin of Wikinerds.org Portal which was slashdotted on 4 February 2005. My site uses Drupal 4.5.2.

According to the usage statistics, at least 17111 people (18622 unique IP addresses) accessed the site, but they may be many more. Currently they have used more than 1.5GB of bandwidth, but they continue to come. However, I have got enough bandwidth from my provider so I think I won't have any problem with that.

My hard disk usage suddenly increased due to various log files: The Apache access log is now 53MB and the referrer log about 16MB.

According to AWStats I have got 215027 hits (not including my usage, various crawlers and bots) and the site served more than 24833 html/php pages. The specific page which is the slashdotted article got viewed more than 19546 times. Webalizer, however, says that I had just 48591 hits. I suspect that due to the configuration of my server the programs may have lost significant parts of the Apache access log.

The heavy usage increased much more after 7AM (USA Utah time - where the server is, far away from me in Europe!).

The throttle module was enabled eventually, but this didn't help my site: At the beginning it became extremely slow and then it started to generate time-out errors. Later it was not responding at all, but as the slashdot readers moved their attention on other sites, my site went up again and it is working now, but it still under heavy usage.

A warn about anyone using images on slashdotted sites: Try not to put many images and make sure that even your logo is accompanied by some text, because in my experience the images on a slashdotted site just refuse to load, if the site is still alive. On a Drupal site it would be better, for example, to have your site name (which links to the homepage) displayed as text and have just a tiny logo.

From the slashdotting I also gathered interesting statistics about the slashdot readers. I posted some of the statistics on slashdot here, but they are a bit old. With the new data, I understand that about 33% of slashdot readers use GNU/Linux and 60% of them use the Mozilla Firefox webrowser (btw I use the wonderful Konqueror on SuSE Linux).

Most slashdot readers come from USA but also UK, Canada and Germany. From the stats I understand that most slashdotters just visit a page and then return to slashdot to post comments. Few of them seem to have visited other pages on my site or even the homepage.

My Drupal log is full of 404 and other errors. Any idea why/how these errors are generated?

--
NSK, Admin of Drupal-based site http://www.wikinerds.org

madpenguin’s picture

madpenguin commented

Our new site, SlackersBible.org was slashdotted a week or so after its launch. Other than an overlooked connection limit in MySQL, the site held up just fine (Drupal 4.5.2). No complaints there at all, and as a matter of fact we were all very very relieved :) Of course we use caching/throttling so I'm sure it had something to do with stability.

nsk’s picture

nsk commented

what ISP do u use?

if u have any 88x31 button I can include it on my site somewhere, for promotion. I think your project is very cool

--
NSK, Admin of Drupal-based site http://www.wikinerds.org

madpenguin’s picture

madpenguin commented

re ISP: we run our own servers.
re button: I sent a copy to you in email. Thanks for linking to our site! That's awesome. Its coming along pretty well so more traffic will definitely not hurt ;)

nsk’s picture

nsk commented

I am looking for cheap dedicated servers. If you have any good suggestion email me.

--
NSK, Admin of Drupal-based site http://www.wikinerds.org

kbahey’s picture

kbahey commented

http://drupal.org itself (this site!) has been slashdotted in the past year or so.

I think it happened more than once.

I did a quick search, but could not locate the article on Drupal.org.

Here are the articles from Slashdot though
http://slashdot.org/article.pl?sid=03/09/30/170259&tid=117&tid=185&tid=103

http://ask.slashdot.org/article.pl?sid=03/11/25/2257243&tid=185&tid=4

http://developers.slashdot.org/article.pl?sid=04/10/24/0113227&tid=146&t...

The first one sure did result in slashdotting. Not sure about the other two.

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

ezheidtmann’s picture

ezheidtmann commented

Debian Planet is currently on Slashdot's front page, but it seems to be holding up well.

http://linux.slashdot.org/linux/05/02/07/0218225.shtml?tid=90&tid=117&ti...

smazsyr’s picture

smazsyr commented

My site, Science Blog is currently being Slashdotted. Other than the server load and memory use briefly jumping, the site has handled it beautifully. I have caching and throttle active, and I think that may help.

Regrettably, I switched servers recently and Google News, previously a top source of traffic, has had trouble updating our stuff. My theory is that they're pointing to the old IP address with their bots. All other Google bots, though, have managed to make the leap. Suggestions?

carl ditzler’s picture

carl ditzler commented

again Science Blog

kbahey’s picture

kbahey commented

http://kerneltrap.org runs Drupal, and regularly comes under the Slashdot effect.

Let us here from Jeremy about it.

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

jeremy’s picture

jeremy commented

KernelTrap started going through growing pains several years ago, before even migrating to Drupal. Originally using PostNuke, a link from Slashdot would bring the site down for most of the day. During my search for something better, Dries approached me and I quickly decided that Drupal was an excellent choice.

Though the switch helped (especially due to Drupal's built in cache), it still wasn't enough. Here's an old paper (PDF, May 2002) that discusses one such Slashdotting early in my usage of Drupal. It was these occasional Slashdottings that lead to my development of the throttle module to automatically disable some functionality when the server load was too busy.

Since that time, I've constantly tuned my server for improved performance. One significantly noticeable improvement came from using the php cache, Turck mmCache. An even bigger improvement came from moving to a dedicated server, allowing me complete control over the software allowing optimal configurations. Keeping up with the latest Drupal releases has helped as well, as each new release tends to offer better performing code.

These days, our average traffic (319,360 hits transferring over 4 GB of data to users viewing 193,712 pages per day) is greater than the Slashdotting discussed in the formentioned paper from May of 2002. A link from Slashdot no longer slows the site down, even a fairly big one such as came with our interview with Richard Stallman posted in January of 2005 (794,609 hits transferring over 11 GB of data to 42,528 individuals viewing 317,436 pages).

james51’s picture

james51 commented

When you use Turck mmCache, do you encode all of the files or do you just load the module into php and leave it at that.

jeremy’s picture

jeremy commented

As described here, I just load the module and leave it at that.

crunchywelch’s picture

crunchywelch commented

mmcache is no longer under active development, but has continued on in another project called eaccelerator. The homepage is here http://eaccelerator.net/HomeUk

I run this on all my servers, I assume it has some performance improvements over mmcache, but for sure is an improvement over not having either ;)

bertboerland’s picture

bertboerland commented

I think kerneltrap is bi monthly or so on slash. once again the site is listed whele remaining extreme fast!
--
groets
bertb

--
groets
bert boerland

jeremy’s picture

jeremy commented

The site had some issues. Two problems. 1) mail_archive is not optimized at all, and consumes a significant amount of CPU when enabled. 2) banner.module has some locking issues that cause it to be really slow under a heavy load.

I ended up disabling my banners for about 3-4 hours under the heaviest load, and the mail_archive for about 2 hours.

Oviously the time has come to fix both of these issues.... ;)

jeremy’s picture

jeremy commented

The banner module has been newly optimized and proven to work under a couple of Slashdottings now. The new system provides the ability to use multiple cache files, rather than having one as a bottleneck. KernelTrap is currently configured to use 10 such cache files, but I've never seen it need more than 5 at a time, even under a direct slashdotting.

Next, to optimize the mail archive module...

kbahey’s picture

kbahey commented

As I type this, LinuxJournal.com is on Slashdot's front page, because of this story about a Linux powered cat feeder.

LJ is not responding at all. The Google cache page is here.

I wonder if they have cache and throttle turned on.

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

bertboerland’s picture

bertboerland commented

spreadfirefox has been on slash for a couple of times and with the 25 million downloads in 99 days, it is on slash again. the site is still fast, the first couple of times spreadfirefox was on slashdot there were some issues with the referral program if memory serves me right 9and dries even helped them out). now all spreadfirefox needs is the clean url option :-)

--
groets
bertb

--
groets
bert boerland

krsh’s picture

krsh commented

Excuse my ignorance but whats stopping you from using the clean url option?

bertboerland’s picture

bertboerland commented

I wish I had anything to do with SFX. But I dont. And I dont know either why "they" didnt use the clean URL option of drupal, they should.
--
groets
bertb

--
groets
bert boerland

carl ditzler’s picture

carl ditzler commented

again and handling it very well.

bertboerland’s picture

bertboerland commented

again, would love to see some stats about the /. effect--
groets
bertb

--
groets
bert boerland

carl ditzler’s picture

carl ditzler commented

Science Blog is /.ed , doing well.

bertboerland’s picture

bertboerland commented

right now we are on the frontpage of slashdot again. I posted "the missing link", there was no link towards drupal the site. More attention for drupal once again and a nice new high profile drupal site... not accessable right now due to all the /.-ers. And mirrordot has a copy but without a stylesheet it seems. --
groets
bertb

--
groets
bert boerland

kbahey’s picture

kbahey commented

The article has a link to OurMedia web site.

It is back up now, but was completely dead for a while.

Did they have caching and throttling disabled, then turned it one when they noticed?
--
Consulting: 2bits.com
Personal: Baheyeldin.com

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

kbahey’s picture

kbahey commented

I submitted the news about Bayosphere and Dan Gillmor using Drupal.

It is on Slashdot as I type this.

Would be interesting to see the slashdotting as it happens as well as post-mortems from both sites.

--
Consulting: 2bits.com
Personal: Baheyeldin.com

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba