Linode.com Forum

caker · Posted: Tue Oct 27, 2009 3:32 pm Post subject: Host Reboots - October 27th, 2009

During a shared library update distributed to our hosts, a number of the hosts incorrectly have marked Linodes as being shut down. To recover from this we may be issuing host reboots to upgrade their software to our latest stack, and then bringing the Linodes to their last state. We're working on this now and expect to have additional updates shortly. We'll also be notifying those affected via our support ticket system. Please stand by.

-Chris

pparadis · Linode.com Staff Joined: 23 Apr 2009 Posts: 24

We believe we have addressed the core issues surrounding today's service outage, and we're working with all customers who are still facing problems to restore their Linodes to full service. We encourage anyone who may still be experiencing problems to contact us via our support ticket system or #linode on IRC.

We would like to take this opportunity to once again sincerely apologize for any issues this outage has caused, and to restate our commitment to handling problems of any kind in as expedient a fashion as possible. As both a company and individuals, we value our relationship with our customers, and we understand the frustration many of you have felt while the underlying causes and resulting effects of this outage have been investigated and handled.

As the situation evolves, we will be conducting an in-depth analysis of the factors that led up to the outage, and developing a solid set of procedures designed to prevent the future recurrence of such issues. Please expect additional information, to be made available once we have thoroughly reviewed the situation. Again, thank you for your support and continued business.

davew · Newbie Joined: 27 Oct 2009 Posts: 3

Before any gripes can be said, let me just thank you guys for the great service you provide and working so hard to get things up and running again. I've been a customer here for a few years now and this is the first time I've had an issue.

Keep up the good work!

roopesh · Senior Newbie Joined: 13 Jun 2009 Posts: 11

Thanks for the update. I would ask that you add a communication plan to your processes and procedures to use in future incidents.

Thanks for the hard work.

gnummep-martin · Senior Newbie Joined: 31 Jul 2009 Posts: 12

mibbit · Newbie Joined: 26 May 2009 Posts: 3

Thanks for the update. Number one request from me would be an alert email to say each of my linodes has been rebooted. *very* critical information that I need to know.

gnummep-martin · Senior Newbie Joined: 31 Jul 2009 Posts: 12

RTBarnard · Joined: 28 Oct 2009 Posts: 1

Thanks for keeping the lines of communication open on the forum and on the IRC channel! You guys really do a great job, and your handling of tonight's unfortunate series of events was really top notch.

When I first noticed that my node was down, I put in a support ticket and had a response explaining the situation and referring me to the forums / IRC channel within minutes. And then when my node still was having trouble after the problem had been identified and fixed for most people, I was able to get one-on-one help right away in #linode from `mikegrb.' The problem wasn't easy to fix, but Mike stayed with me for over two hours as we worked out the solution together.

Having interactive one-on-one help like that isn't something many other hosting companies have offered me in the past, and those that have usually abandon me if the problem isn't solved in the first 5-10 minutes.

If problems like these were rampant, and I found myself needing technical support on a regular basis, I'd probably be singing a different tune right now, but with this as the first unexpected downtime in over 2 years of service, I have to say I'm really pleased by Linode's response. Thanks, Linode!

bd3521 · Senior Newbie Joined: 05 Feb 2009 Posts: 11

oliver · Senior Member Joined: 02 Apr 2008 Posts: 90

mnordhoff · Senior Member Joined: 03 May 2008 Posts: 179

oliver · Senior Member Joined: 02 Apr 2008 Posts: 90

dmuth · Posted: Wed Oct 28, 2009 9:33 am Post subject:

Suggestion for your communication next outage.

What Google did during their last outage was to post an update, followed by the text, "We will update this message again by 5:19 PM", where the time given was an hour in the future. That way, people wouldn't be refreshing the forums every 5 minutes, but could busy themselves with something else for the next hour.
_________________
Disclaimer: I am not an Linode staff member.

gsf · Posted: Wed Oct 28, 2009 9:59 am Post subject: Another communication suggestion

Dear Linode staff,

See http://blog.bitbucket.org/2009/10/04/on-our-extended-downtime-amazon-and-whats-coming/ for a pretty well written post-mortem.

eas · Junior Member Joined: 21 Mar 2009 Posts: 34

This outage was frustrating, but thankfully 1) unusual & 2)relatively brief (for me, at least). I'm glad to hear that you'll be doing a post mortem and sharing your conclusions. I look forward to what you find. I hope you will consider releasing interim findings and conclusions along the way.

In addition to the technical/operational issues that led to this outage, I hope you'll also tackle communications issues, both with proactive warnings of potential downtime, and also with multichannel communications with customers when linode's own infrastructure is compromised. The first thing I did after trying to ping and ssh to our offline host was visit the linode homepage and find that it wasn't available. I ended up thinking to look to Twitter at about the same time your pages started loading (slowly) again. I checked in on IRC once I could find the server information. While your site was offline, it would have been great to have gotten an email, or to be able to register to get SMS notifications.