Black Wednesday
Blog calendar
— or —
Blog tags
about
activity
ads
amazon
android
aws
barcamp
barcamp.pl
blogging
canvas
cdn
cf
chef
cloud
cloudfront
design
dev
devcamp
ec2
email
firmware
free
fun
funny
gadget
gadgets
game
gtd
haproxy
howto
html5
ideas
ie6
insider
iphone
itunes
jailbreak
javascript
kids
leaks
marketing
meetings
mobile
monetization
mozilla
notifications
outage
phone
php
picture
05 Mar 2009 09:21
There are weeks that nothing exciting (or fatal) happens. But there are days that a lot happens that make you think if this is really a coincidence. Yesterday was one of such days, and it was not even Friday 13th. So here is what happened:
- I stayed at home, Lukasz was at the office. All of sudden network went down. Later we learned that half of our city (that get internet from TP S.A.) had problems.
- Later Piotr called that our office server is down and cannot boot up. It was one of the disks in RAID 10 array that failed and for some reason GRUB could not boot. It booted later after Piotr did some magic, now we just need to replace one drive asap.
- At 15.30 local time I got an alert email that Wikidot.com is down. Immediately i tried to log-in to the server - nothing. Ping - yes. Alive. But all services went down.
- After a few minutes we knew we must act. Piotr started re-assigning IP addresses of the web server to a backup server. Failed. Looks like the router could not handle this in real-time this time.
- Main server restart - nothing helps. We had a similar issue some time ago, we started the rescue mode (server boots from a rescue linux image, this is greatly automated by SoftLayer). Server is up. A year ago what prevented the system from booting was a forced fsck on one of the drives and this required a key pressed or so (as told by the SoftLayer support team). So we started disk checks. And this took almost an hour! S#*t!
- Meanwhile my friend called me as his car broke just 20 meters from our parking lot and he could not move it, so I went to help him.
- Server got up, everything was back to normal. Situation under control.
I am not afraid of fatal Fridays any more. I fear of Wednesdays.
rating: 1, tags:
You really should study physics and the laws of science. These laws are unbreakable, you know. If you studied physics, then you'd remember Murphy's Law — Anything that can possibly go wrong, will. =D
From this law of physics, I can deduct right here and now what will happen when you finally finish Wikidot 2.0. Within the first hour of launching it, the servers, and all cloud-front servers, will be attacked by terrorists with bombs. The data will be irrecoverable. Simultaneously there will be a critical fault with the backup systems containing the alleged 2.0 platform, and the backup servers will burst into flames and the data will be irrecoverable.
Also, it will so happen that you will be away from all of this on a holiday in Australia, and you will live to think of the thousands of hours of investment in the revolutionary wiki engine.
Murphy's Law… I don't wish it to happen, it will happen =/
Black Wednesday recalls me that Microsoft release some "updates" to their "operating system" on Tuesdays. They are called then Patch_Tuesdays. Some though prefer to call them Black Tuesdays or even better Blue Tuesdays because many computers crash (and display BSOD) after applying the "updates".
Let me be clear: WE DON'T USE ANY MICROSOFT SOFTWARE ON OUR SERVER!
Piotr Gabryjeluk
visit my blog
Coincidence does not mean causality. Sometimes things just happen. Luckily the problems were resolved and you got the inspiration for a blog post :)
I think I traced this. Server problem. It was tough. I made some precaution changes to our configuration and I will post more details once I can confirm my findings.
Looks like a terrible bug in PHP FastCGI interface that can lead to a "self-DOS".
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me
Thanks for sharing this nice article. I read it completely and get some interesting knowledge from this. I again thanks for sharing such a nice blog.