After what can only be described as the 'Extended Weekend from Hell', Drupal.org has been restored and is back in business.

First things first: what caused this outage, and why did it take so long to get the server back up? To phrase it simply and bluntly: misfortune, bad timing and miscommunication. Murphy's (annoying) law has been proven true once again.

At the moment, Drupal.org shares its resources with several other sites on a server kindly provided by Kjartan. Thursday evening, this server was hacked. One of the other sites on our server provided the hole through which the hackers entered; it appears someone wanted to turn us into a warez FTP, but completely messed it up instead. We discovered the intrusion quickly and were able to regain control of the server soon afterwards. However, the entire incident occurred only a few hours before a scheduled power outage at our current ISP; problems with remote administration and the lack of install media meant we were unable to fix the server remotely. Over the weekend we called to try and rectify the situation, but due to miscommunication with our ISP we had to wait until Monday morning before we could reinstall the OS and get the server purring again.

Still, it is not our intent to sling mud and point fingers. If anything, the whole experience has been a lesson for everyone involved.

Now, as you could read on the temporary page during the weekend, we were already planning to get another hosting, one dedicated to Drupal and related projects. Talks have been going on with Scott Kveton of the Open Source Lab for several weeks: they will provide everything we need for hosting and support, as long as we provide the machines. For this we planned a fundraising event which took place this weekend. The fundraiser was a great success, and during the next few weeks we will be gradually migrating to the new servers and new hosting facility at OSL!

As the amount you managed to raise for Drupal was above and beyond our expectations, we have asked the OSL to come up with a complete infrastructure plan for Drupal.org and its related services which will optimize reliability. Rest assured that your money will be well spent. Stay tuned for more information about this.

Comments

Steve Dondley’s picture

Actually, I think the downtime for Drupal paid off handsomely. How much was raised?

Dries’s picture

We'll follow up on that shortly.

mattm’s picture

Was this other site which provided the hole an unpatched drupal?

bertboerland’s picture

the changes that "these" people run unpached drupal is next to nill, running apache/php with root access would be rather silly. most likely an ftp/ssh that was misused?

either way, i am sure dries (or another) will file an abuse at the (also hacked?) source ip address.
--
groets
bertb

--
groets
bert boerland

jamiejamie’s picture

Talk about panic, lol.

I visited the site to download some modules and found it dysfunctional. I immediately began a quest to gather the nickles, dimes and quarters that have been collecting in the ash tray of my car, so that I could make a donation.

Get a good server. We are counting on you Drupal.org, and thank you for developing this fine product!

Jamie


http://www.jamiejamie.com

Morbus Iff’s picture

The word you want is "cracker", not "hacker". A hacker wouldn't want to setup a warez FTP site.

http://disobey.com/
http://gamegrene.com/

Steven’s picture

I'm in the "context decides its meaning" camp... I think everyone understood the message. Geeks seem to have the most trouble with the idea that words can mean multiple things :P. And of course, it's pretty hard to resist the indoctrination by mass-media.

--
If you have a problem, please search before posting a question.

Jason_xo’s picture

I hate to burst your bubble, but a cracker reverse engineers software. Nothing to do with [b]hack[/b]ing in to sites, it would infact, be a hacker as already stated.

Toe’s picture

I'm in the "Who gives a flip?" camp. :P

Kobus’s picture

I don't care what you call them. You can call them "idiots" too, fine by me!

-- Kobus

Bèr Kessels’s picture

h@x0rZ vs hackers? I am sure eveyone knows what the first one means :)

---
if you dont like the choices being made for you, you should start making your own.
---
[Bèr Kessels | Drupal services www.webschuur.com]

voipfc’s picture

my blog suffered a lonely weekend :-)

kbahey’s picture

I happened to have a few web browser screen open when donations were coming in.

I summarized what I could find here: Drupal Open Source community exceeds target in fundraising for new server, which gives a glimpse of how things progressed, and the potential role of Slashdot posting a story on it.

With that much money, we could pay for the legal fees to make a Drupal Foundation a reality, we could run two servers as mirrors, separate MySQL from Apache/PHP, ...etc.

--
Drupal development and customization: 2bits.com
Personal: Baheyeldin.com

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

sepeck’s picture

Keep in mind that Dries did the updates by hand manually. It will be interesting to see, and although slashdot probably had some impact, I suspect that moneywise we will find that the bulk of it is from within the Druapl community iteslf.

We asked for funds for hardware. We are probably obligated to spend it on hardware. People might get upset at anything else. :)

OSL is working up a hardware architecture for the setup, but servers have been ordered, they will take a few days to get shipped, not much we can do about that. Dries will post a more detailed update soon but remember, some folks have paying jobs they have to go do in the meantime or paying for rent and food gets hard to come by.

Remember, Friday it started, access to rebuild was only on Monday for a fairly complex server recovery with multiple sites, mail, listserv, cvs, automation scripts to recover. Plus, the request for donations was only put up Sunday and was more then anticipated.

So, discussions with OSL were on Monday noon (They are in Oregan, US so that's Monday PST (GMT-7/8)). Now it's Tuesday and the server is up. Folks need to sleep, go to work, say hi to significant others, etc. Stuff is happening. Now that it is happening, and the server is up. Hardware is on order and soon to be i the mail(well, shipping company). We can all take the time to implement the next hardware build out in a configuration that will last longer and provide more flexibility.

-sp
---------
Test site...always start with a test site.
Drupal Best Practices Guide

-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide

merlinofchaos’s picture

As someone who donated, I would be just as happy if some of that money were also spent creating a drupal foundation. The way the donation request was worded, that seemed to be a good second use for the money once the hardware requirements were meant. I would be surprised if many -- heck, any -- people were upset if money went to do that as well.

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

bertboerland’s picture

infrastructure has a broad definition:

i like:
The basic facilities, services, and installations needed for the functioning of a community or society
--
groets
bertb

--
groets
bert boerland

jesusphreak’s picture

I really don't think anyone would mind their money being used for non-hardware things. A Drupal Foundation would be a definite plus. Anything to benefit Drupal is a plus, not just hardware.

cel4145’s picture

but it's worth saying. If at all possible, the mailing lists should be on a separate server from drupal.org so that *if* the server with drupal.org goes down--routine or unexpected maintenance--communication still exists for the Drupal community. As long as both these resources are hosted on the same server, another complete communication blackout for Drupal is always a risk.

tag’s picture

Perhaps a mirrored status page on some other sites with [separate servers] and the willingness to do so -- rather than a whole separate box for mlists? Maybe some of the companies using Drupal as their main offering? Or some of the core developers' sites [or do they all live on the same server?]. Again, likely already planned, just stating the obvious...

cel4145’s picture

merely on separate boxes. for example, it might be useful to put cvs and mailing lists on one box, drupal installations on another.

suzanne.aldrich’s picture

Were people who have the drupal login module installed affected by the outage? Seems like that would be better off on another box, as well...

rubicon’s picture

When the site went down I did some searching and it seemed went down due to security problems with Comments. Either I read the dates wrong or it was misinformation.

Regardless, it's great to see the site back and the donation amount was 3x your initial asking. Awesome!

iandickson’s picture

I started in Finance.

My comment - any Foundation should be set up under the rules and jurisdiction of the country with whom those will run the foundation are based/familiar.

Anything else will enrich lawyers rather more than they deserve.

So prob looking at a Netherlands NFP or Trust type arrangement.

(Other Foundations in other jurisdictions can be set up if it becomes clear that people those jurisdictions are going to pony up sufficient donations to make it worthwhile.)

Jonathan Furness’s picture

Well done for bringing back Drupal.org... we missed the site over the weekend. I';ve experienced similar... and you are spot on with saying that it has been a learning experience.

If, like me, this will allow you to put processes and procedures into place that will safe guard, to some extent, a similar experience re-occuring. Keep up the good work,

Jonathan

Jonathan Furness
teacher, developer, webmaster
http://www.jonathansblog.net

Eagle-i’s picture

One of our drupal sites was hacked last week but we were able to restore it. It looked like a hacker from the Netherlands but that url was probably hacked itself. Does this point out some vulnerabilities though? Not sure if the site had the latest version but now it does.

Hope Drupal will get a real positive boost from this experience.

___________________________________
Building (drupal) webblog portals & communities

laura s’s picture

I've not looked at the installation instructions in a long long time, but it would seem helpful to have IN BIG CAPS strong recommendations for the admin to sign up with the email newsletter in order to get the latest update announcements. How many admins of Drupal-powered sites out there still have no idea of anything that's happened in the past 2 weeks?
===
Laura
pingV

_____ ____ ___ __ _ _
Laura Scott :: design » blog » tweet

robertDouglass’s picture

All I can say is, assume you've been attacked. If you're not running the latest and you're on the internet with your site, they've found you and have at least tried to take over your machine. The bad part is, you won't notice. They're not going to send you an email or post an alert on your site -- but they're there.

- Robert Douglass

-----
If this helped you, please take the time to rate the value of this post: http://rate.affero.net/robertDouglass/

www.hornroller.com, www.robshouse.net

my Drupal book | Twitter | Director, Product Operations Commerce Guys

laura s’s picture

...but many people do not do that. To a large extent, people are accustomed to software telling them when there's an important update. Or, as part of some registration process, they are automatically notified when a security issue is released.

And it might just slip people's minds. They come here, download, install, ask questions, fiddle for a while ... and then other things come up, and they don't have the time or ongoing developmental interest to keep coming back to Drupal.org every day.

And signing up for a security issues newsletter may not occur to them. Many may not know it's possible. So a simple urgent reminder in the installation instructions would give them all they need. Something like:

"Be sure to sign up for the security issues newsletter [link]. This email list is used only for announcements of new releases of core software. Do it now, before you forget."

===
Laura
pingV

_____ ____ ___ __ _ _
Laura Scott :: design » blog » tweet

sepeck’s picture

Maybe we can work on some verbage (SHORT) to add to the Install.txt on this and submit a patch for it. Something like

Be sure to sign up to the Drupal Newsletter to keep updated on the latest security related issues and general Drupal news. Please read the Handbook Admin guide and best practices section for more tips.

-sp
---------
Test site...always start with a test site.
Drupal Best Practices Guide

-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide

Dries’s picture

Good idea

sepeck’s picture

I'll add some Security Basics stuff to the Best Practices section this weekend and include that suggestion. Good call.

-sp
---------
Test site...always start with a test site.
Drupal Best Practices Guide

-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide