By Steven on

You may have noticed that the Drupal website has been slow lately. Since the 4.5.0 release, Drupal has gotten a large increase in popularity (more than 10.000 visitors per day worth 70GB of traffic/month). This, coupled with the heavier theme and an aging server has been the source of the performance issues.

Kjartan is busy moving Drupal.org to a new server and host. The old server (a four year old Pentium 3 1Ghz) was not only running the Drupal.org website, but also other sites as well as the Drupal mailing lists, archives and CVS repositories. Our new server (a Pentium Xeon 3Ghz) should be able to handle all those things much better. We'll be moving over to the new server as soon as possible, hopefully before the weekend. Because we are going to switch DNS servers shortly, the site, mailing lists and the CVS repositories might become unavailable while the DNS changes propagate.

On the bright side, the stressed server has allowed us to further analyse Drupal's performance: some performance enhancements already made it to the Drupal code, others are still being worked on.

All this prompted us to think about Drupal's future. We figured that some form of Drupal foundation might be useful. The foundation's purpose would be to manage resources for Drupal: collecting funds, taking care of hosting costs, promoting Drupal, etc. At this point, we are merely investigating the available options, as well as collecting advice.

For now, we have installed a Paypal donation button on the website. The money we receive will be used for supporting and growing the Drupal project. For example, for LinuxTag 2004, the Drupal team designed promotional brochures which were handed out to visitors there. Getting the brochures done involved a lot of overhead in printing and distribution. And of course the new hardware for beefing up Drupal.org is expensive too. The goal is to see how much interest we get for Drupal and figure out how exactly a foundation could help the Drupal project in the long run.

Comments

fungy’s picture

fungy commented

I've learned the lesson the hard way. Web and db servers doesn't mix well in a high load site. Web connections are blocking and slow, web <-> db connections are fast. Web server is cpu hungry, db is memory hungry. Putting them on one server makes them congest each other.

I run a site (http://isohunt.com/) with 10x Drupal's traffic, and it's running well with 2 whitebox Athlon 2600 servers. (it's not running Drupal though) Depending on your hosting costs of different hardware, you may get better bang for your buck running 2 cheaper servers than 1 expensive one.

Anyways, my 2c. My host is Serverbeach, I can give you 1/2 price off their hosting packages as my contribution to Drupal.

PS. I'm about to do a lot of hacking on Drupal for some mass Drupal deployments, so I'll contribute whatever performance enhancing code I make. From what I've seen so far, Drupal is superbly designed, so thanks to all the devs!

Chris Johnson’s picture

Chris Johnson commented

Fungy is right. It's always best to run web servers and databases on separate machines whenever possible at our end of the scale, i.e. using commodity PC-style machines.

I've seen a bunch of inefficient code in Drupal. I just have not had time to create patches for them. There is plenty of room for improvement in Drupal's database code in particular.

killes@www.drop.org’s picture

killes@www.drop.org commented

Even if you don't have the time to create patches ou should create issues or send a mail to the mailing list. Maybe somebody else has more time.

--
If you have troubles with a particular contrib project, please consider to file a support request. Thanks.

--
Drupal services
My Drupal services

Dries’s picture

Dries commented

Don't panic when the 'Forum topics' or 'Drupal talk' blocks are gone.

I've enabled the throttle module here on drupal.org. The throttle module provides a mechanism for automatically detecting a surge in incoming traffic and to respond to that in one way or another. So whenever there are too many visitors (either anonymous users or authenticated users), some of the blocks in the sidebar will automatically dissapear for a while. Disabling some blocks reduces the page generation time and helps our poor old server dealing with the traffic.

killes@www.drop.org’s picture

killes@www.drop.org commented

Throttled blocks should display a "not available due to high load" message. Otherwise users will get confused.

--
If you have troubles with a particular contrib project, please consider to file a support request. Thanks.

--
Drupal services
My Drupal services

ax’s picture

ax commented

+1

Jeremy’s picture

Jeremy commented

I have provided patches before that offered this functionality, but it was nixed... I don't remember why offhand, but you can search the -devel archives.

This wouldn't be possible if throttling an entire module.

pamphile’s picture

pamphile commented

Many of us can't use PayPal. PayPal does not accept payments from at least 100 countries... These additional payment processors should add variety:

stormpay ( free signup )
Amazon - ( free signup )
2checkout ( no free signup )
worldpay ( no free signup )

Please add two more donation processors... :)

Thanks
Marcel
http://computingnews.com

harald.walker’s picture

harald.walker commented

I do not agree with eBay's/PayPal's privacy policy and am not going to use their services again.

Amazon would be fine or otherwise give us an international bank acount number (IBAN) to transfer some money directly.

Dries’s picture

Dries commented

I just updated drupal.org's nameservers. And so the swtich began ...

kbahey’s picture

kbahey commented

First, I agree with separate DB server and web server. This should make things much better.

As for the reason for the sluggishness, I am not disputing anything here, just an observation: seems to me that the slowness in Drupal.org coincided with the release of 4.5.0. This made me suspect that it is code related, and not just volume.

How many visitors per day were done before the current surge?

I know that the site was Slashdotted one or two times in the past year, and it was performing well under the load even.

I am trying to avoid a situation where we have a new server, then things are ok for a few days, then sluggish again afterwards, because the underlying cause is something else (e.g. all database on one disk, code bottleneck somewhere, ...etc.)

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

drumm’s picture

drumm
he/him
Location NY, US
commented

Drupal.org was running CVS versions of Drupal before 4.5 was offially released so the difference in running code is minimal.

Steven’s picture

Steven commented

Indeed, Dries is right. Before the release, we updated Drupal.org to 4.5-RC, and found several performance issues. They were fixed, and the site was a lot faster. It was only after 4.5.0 release that the problems got really bad.

Dries’s picture

Dries commented

During the Drupal 4.5 code freeze, I did compare Drupal 4.5's performance to Drupal 4.4's (or something that was close to being Drupal 4.4) and Drupal 4.5 was faster than Drupal 4.4. I shared some of the results on the developers mailing list so you can find the results in the archives.

Maybe someone should compare them again? Maybe I missed something, or maybe I used irrelevant tests. Maybe. Or maybe someone already compared performance of Drupal 4.4 and Drupal 4.5 prior to upgrading his or her site(s). If you did, please share your results and experiences. Thanks.

killes@www.drop.org’s picture

killes@www.drop.org commented

The numbers you produced were a bt skewed due to the newly introduced gzipped cache. If people where using mod_gzip with Drupal 4.4 the change wouldn't be that big. it would be helpfull to re-run the benchmarks with mod_gzip turned on.
--
If you have troubles with a particular contrib project, please consider to file a support request. Thanks.

--
Drupal services
My Drupal services

Dries’s picture

Dries commented

In less than 24 hours, we got 6 donations worth 115 euro in total. Thanks guys, you rock!

Creative ideas on how you'd like to see this money spend (other than on hardware, hosting or promoting Drupal) are always welcomed.

arnabdotorg’s picture

arnabdotorg commented

I think it would be cool to have a "Drupal Bounties" section where people can offer money in exchange for certain optimizations / features/ modules.

It's a great way to attract both funding and developers.Mozilla & GNOME and many have used this model quite successfully, and given the rising popularityof Drupal in the commercial space, I don't see why it shouldn't workwell here.

fungy’s picture

fungy commented

I second that. People get paid for hacking open source code for commercial use. Perfect arrangement to me.

---
IRC and BitTorrent search engine
My first Drupal site

kbahey’s picture

kbahey commented

Great idea arnab!

Users are willing to pay for themes, and features. Why not give the users what they want, and use that as a way to give back to the community?

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

Bèr Kessels’s picture

Bèr Kessels commented

I investigated Paypal, but did not find an easy way to show the "total donated" on ones website.
It would be nice if we can show a line under the [paypal] button showing the amount donated in total since XXXX.

Also, we would really need a page (or more) on the site showing the donamted amounts, the names (privacy?) and times of donations. Also some genreal amounts outgoing, for example:"new server setup | 200 E"

Ideally this would be automated (paypal framework), but for the time being we can do this by hand and update that page once a week.

We also need a page about "donating" just some small notes about where the money is used for, why we need it and more. When its only in this thread it will get lost.

[Ber | Drupal Services webschuur.com]

pamphile’s picture

pamphile commented

Since I live in Trinidad, I can't use PayPal...

arnabdotorg’s picture

arnabdotorg commented

http://www.dropcash.com/

crackerjackmack’s picture

crackerjackmack commented

There is already the paypal framework module in the downloads section. It handles DB and payment integrety with paypal using IPN. The README includes a few small blocks of code on how you can create your own blocks. Minor amounts of php honestly.

As for automating the task of creating the block, I've thought about that honestly, but haven't had the time to write it up. I just got promoted to head of special projects at work (awsome, but I work 60hrs/wk). Which in turn, I've made our internal development site drupal!!!! My boss is ultra (mega even) excited about the changes I've implimented so fast.

Thanks to drupal for this great engine....now to find that donation button...

Dries’s picture

Dries commented

Agreed. We should probably look into the contributed paypal module(s) and see how it can be integrated into drupal.org such that tracking donations is fully automated. Admittedly, I know nothing about Paypal's APIs or these particular modules. Anyone willing to help with this?

Update: I began looking into the contributed paypal modules (ecommerce module, Paypal framework module, Paypal tipjar framework). I haven't tested these yet, but I had a quick look at some of the code.

Dries’s picture

Dries commented

Investigation learned that neither the 'Paypal framework module' nor the 'Paypal tipjar module' are working properly. I filed some bug reports.

Crackerjackmack, the author of the paypal framework module, is already looking into improving/fixing his module. :-)

The tipjar module is no longer being maintained, it seems, and doesn't work with Drupal 4.5.0 or Drupal HEAD.

Boris Mann’s picture

Boris Mann commented

I set up a Drupal store. Shipping costs make it really expensive anywhere outside the US, but I thought some people might like to donate in that way, and have a product in return. I've marked up all the products by $5US, all of which will be donated to Drupal.org.

Steven, I hope you don't mind me using the Druplicon (again!), and if you (or anyone else) send me some other designs, I can post those as well.

y2kbugs159’s picture

y2kbugs159 commented

greeting to DRUPAL!!!!!
thank for the respectful ppl who donate $$$ for the Xeon server.
thank to the developers, all drupal users, who ever like drupal......

i've just try using drupal, it is a GREAT CMS ever......

i'll keep promoting DRUPAL to everyone!!!!! ^^

gkrishna’s picture

gkrishna commented

It's nice to see the site back. Well, i know many "drupalians" would have felt restless for the past two days. Ofcourse, drupal is getting better and better, more matured.

A fantastic community portal.

--**--
krishna

Farms’s picture

Farms commented

As a new user I'm falling deeper and deeper for this :o)