Community site broken again

Assuming this isn't an April Fool's day joke, I'll try to take a look this afternoon. If not, tomorrow morning.

To my knowledge the site hasn't been upgraded recently.

The site does have been upgraded to 1.3 and is broken now. Who did that without testing on the cloned copy first? :/

Alright people, it's not an april fool's, and we're sorry, but there was an upgrade to 1.3 that screwed up.

We are working on restoring from backups as we speak, hold on.

So things look grim: it looks like we have a bug in our migrate code that drops the comments table on that site (?!!).

This, coupled with the fact that we have a failure of incremental backups on the mysql server, means that we have lost 3 months worth of comments on the site. Other sections of the site may be affected we haven't found out about yet.

We are working on restoring the site to the best we can, but expect some data loss.

Alright, the site is restored, but we did loose about 220 comments.

I will now post an announcement on the community site itself.

Here is the post: http://community.aegirproject.org/discuss/data-loss-during-failed-upgrad...

We were able to find another backup, and so we're down to only about 20 lost comments.

This site was upgraded by mistake *again*. So back to square one. :(

For the record, koumbit's tracking number for this is #116340. If it happens again, write support@koumbit.org.

No need to do so right now, I know this is broken and will fix it.

FYI: here is a full static copy created earlier today: http://cao.aegir.cc/dashboard.html

it's back, although we have lost today's edits. if there is anything people need, i probably have it somewhere, unless it's comments in which case it's just gone again. :(

The guy who did it again should use my archived copy to restore lost content and comments, as 90% of them are available here: http://cao.aegir.cc/index.html

as far as i know, most content on the static site is also on the prod site now, if not i can restore from the other backup.

let me know.

No, it isn't, just look at the dashboard on both sites. There is no content (on the restored live site) added/modified on April 12 at all.

I already restored two important comments (one really big) yesterday so I don't plan to work again on something broken again. It is an epic fail, btw.

As an example: on http://community.aegirproject.org/discuss/wheres-aegir-drupalcon-denver there is missing the comment I already restored yesterday, plus also a big response by ergonlogic is missing there (but I have an e-mail copy).

The point is that it is a job of those who managed to break the site twice in a row.

I think we did our job: we restored from the backups we had, with the information we had. We also happen to have been hosting the site and mailing lists all this time without complaints, so I am not sure we desserve the flak we are receiving here right now.

Now you come around and give us more information, and more data to restore, and we are thankful for that, but I fail to see how bashing us for "breaking the site twice in a row" is in any way productive.

In this post I have already requested people to provide us with data they feel is missing. You even went in and provided extra content, which is great, but it otherwise doesn't feel like much is missing anymore.

So I do not feel there is anything left to do here. Note that our staff was instructed to not upgrade this site without first doing proper testing and db-level backups first.

By the way, if anyone wants to actually help the underlying issue, it seems we need to port that funky discussion feature, thanks to mig5 for the heads up there. I have pushed the alleged guilty code here: git://git.koumbit.net/atrium-discussions.git See also https://redmine.koumbit.net/projects/atrium-discussions - which could be pushed to drupal.org if it's really relevant.

Finally, understand that I am angry and frustrated by this as you are. This hurts our reputation and we will do our best for this to not happen in the future...

I was very angry first, but then decided to help instead of just express my frustration.

Thank you very much for your help grace, it's really appreciated and means a lot to me.

Thanks everyone for your support, hopefully this will never happen again... :/