Linode Updates for Major Issues is Not Very Good

If you're on london814 I just asked support and they said there's no estimate on how long the transplant will take so it's a case of twiddling thumbs.

Yes Linode814.

But thats the problem… why is no one able to give an estimate? Have the data centre not given Linode an estimate? If not, is it because they have not started? Perhaps there's a big issue there, so a big queue, lots of hardware to fix. If they have started, they should be able to estimate how long a hard drive transplant should take? Are they intentionally not telling us/Linode the issue because its serious? Its very odd no one can give an update on the status.

And thats the point of the post, to give my opinion that updates are not very good.

I do not accept we should be twiddling thumbs, I believe someone somewhere is able to give a better update with estimated timing. Once we know, we can stop monitoring the situation, inform customers and get on with other things in the meantime. Instead I am refreshing the support ticket page every minute for updates.

You could just keep poking support and ask them instead of asking here. Support don't read the forums very often. Anyway they've said the drives have been transplanted and it should be back to normal soon.

@amityweb:

… I have no updates to pass to my customers who services are down.

I agree that the trouble ticket/status updates should be clearer and more forthcoming, but…., are you selling guaranteed uptime to your clients? If so, then you should have safeguards in place for unexpected, and beyond your control, events. If not, then your clients shouldn't be complaining, or if they do, you need to point out that they are not buying guaranteed uptime (which is usually a magnitude or higher more expensive).

They're coming up now :)

I've been asking them for updates, I was not asking for updates here, but just reporting that I don't think Linodes updates are very good. Wondered if anyone else agrees or not, or just happy to be in the dark when servers are down.

I thought if a lot of people agreed then maybe its something they should sort out. If no one complains like this, they'll just keep not updating us.

> If so, then you should have safeguards in place for unexpected, and beyond your control, events

I'm not running Google, sadly we do not have the resources to invest in such an infrastructure. We have one mail server and do not have a backup solution to kick in when the main server goes down (we of course have backup but not a mirror server so users mail keeps on working with no disruption). Would be great but not there yet.

So the post is not about the fact there was an issue, I accept there will be hardware issues. Its about not receiving very good communication on the matter. The issue was fixed 15 minutes after they told me they dont have any timing to give which is quite odd they cant see they are near completion at that time.

Maybe Linode should do a Scotty and make an estimate then multiply it by 4 so they have a reputation of miracle workers when they fix things!

If I get an estimate that's 4 times longer then what they actually take I will be happier than not having any estimate at all.

To suggest not to use timings on things in case of missing deadlines is ridiculous and not very professional. I would prefer an estimate that is missed, than no estimate at all. At least it gives us a guide. For example, lets say they say it would take approx. 3 hours, well if takes 4 hours then thats better than not being told anything because for all I know it could take days. I have no idea if its 3 hours, 3 days, 3 weeks, or unfixable. Saying "several hours" would be a big help. And I would say they must have an idea of this being "experts", they must have done this before? They must know if it will be within an hour, within several hours, within a day, within a week?

@amityweb:

To suggest not to use timings on things in case of missing deadlines is ridiculous and not very professional. I would prefer an estimate that is missed, than no estimate at all.

Would you prefer an estimate that is made up to none at all? Because that seems to be the logical conclusion here.

I dont know what you mean. I give time estimates all the time to customers, but I dont know the exact timing. But I need time estimates so they can know when to expect it and for costing purposes. Maybe a customer has an exhibition in a few days and he wants to know will it be done by then. So I tell him yes or no. I dont tell him I have no idea I will just get on with it and you can wait and see.

I guess you've ordered from Amazon and you like the fact it states delivery estimate between 23rd and 26th September or something, and would be quite annoyed if it said "unknown delivery date, just wait for it".

Decisions can be made with estimates. If they said a few hours I will wait, if they said a few days I would create a new Linode and restore my backup to it. What do you do when no timing is given? Then 3 days later they're still down and you wished you just restored the backup to a new Linode 3 days ago.

I honestly think expecting some estimates on timing is a very reasonable request, and it amazes me people disagree and think no updates on issue like this is acceptable.

@amityweb:

They must know if it will be within an hour, within several hours, within a day, within a week?
You've obviously never worked on system hardware before. Often until you're knee deep into the problem you have no clue what is wrong or what the fix is. I'd say the standard practice is to fix the problem, and it will be done when it's done. Guessing is a waste of time and just upsets the unwashed hordes when the guess doesn't magically convert to fact. I feel your pain, I just endured a city wide power outage for over 4 hours a few days ago. DMEA's website (the electric company) outage status widget just kept saying "repair in progress", no estimate, no update, until the lights came on, then 15 minutes or so after that, the status turned to "repair complete". The trick to managing the unknown is for YOU to have your own time limits and responses. After a certain amount of time, send out alerts, after a bit more time redirect traffic to a status page, after still more time, move all sites to a backup server in another datacenter, etc etc. You can control your actions, you can't control the time it takes for an unexpected event out of your control to be fixed.

@vonskippy:

the standard practice is to fix the problem, and it will be done when it's done. Guessing is a waste of time and just upsets the unwashed hordes when the guess doesn't magically convert to fact.

^ This.

Thanks for the feedback.

I just wanted to sing the praises of the Linode LLC admins and support guys for their excellent jobs when recovering from hardware problems. This happens night and day, weekends, holidays - year round, non stop. Downtime sucks, yes - but these guys do an amazing job getting Linodes back online as safely and as quickly as possible, in spite of this being a frequently thankless job.

-Chris

@amityweb:

I dont know what you mean. I give time estimates all the time to customers, but I dont know the exact timing. But I need time estimates so they can know when to expect it and for costing purposes. Maybe a customer has an exhibition in a few days and he wants to know will it be done by then. So I tell him yes or no. I dont tell him I have no idea I will just get on with it and you can wait and see.

I guess you've ordered from Amazon and you like the fact it states delivery estimate between 23rd and 26th September or something, and would be quite annoyed if it said "unknown delivery date, just wait for it".

Decisions can be made with estimates. If they said a few hours I will wait, if they said a few days I would create a new Linode and restore my backup to it. What do you do when no timing is given? Then 3 days later they're still down and you wished you just restored the backup to a new Linode 3 days ago.

I honestly think expecting some estimates on timing is a very reasonable request, and it amazes me people disagree and think no updates on issue like this is acceptable.

Have you ever worked on a hardware or software outage? I'm going to guess not, because your examples are an apples-to-oranges comparison to an outage.

You building something for a customer can be estimated based on your past experience. Amazon shipping something to you can be estimated based on the distance covered and carrier involved (eg USPS takes longer than UPS or FedEx).

Outages don't work like that. They're all different. Can certain pieces be estimated based on the past? Sure, but what if something that worked in the past doesn't work this time?

If there is no estimate, it's because we expect it to not be long. Depending on the issue that could be 15 minutes or an hour or so. If we know it's going to be a longer haul, we'll tell you. If we know there's a delay, we'll tell you. If we have information that will allow you to make more informed decisions, we will tell you. Otherwise, assume "not long".

If it's helpful, I could provide you with some rough ballpark times for various events. But, we've done this for long enough and enough times to know that a transplant takes anywhere from 20 minutes to 90 - depending on complexities, circumstances, safety checks, etc.

-Chris

So estimates are known then. This information would have been really helpful in my support ticket, this is what I mean by updates are bad, you knew the above info but didn't tell me. Knowing that no timing means possibly not too long is good so now I know this for future issues. Thank you.

@vonskippy My background is not sysadmin, it's engineering and website development. So I am very used to time estimates. But even a sysadmin amateur like me can give certain estimates to customers like how long a new Linode would take to setup and configure. Because I've done it before. I would have expected the data centre engineers to have transplanted a hard drive before so know roughly how long it takes (which now we know that they did know). If they've not done it before and so don't know the time, then tell me that!!

A new concern I have is no one seems to care no time estimates are given - even if they are roughly known. It means the issue is unlikely to be addressed in future. I would have thought everyone would like to know time estimates so I am amazed people don't. The quality of service won't improve without the demand of customers. This could result in a slipping of service quality as competitors do improve such things. It's very important service quality is continuously improved. My engineering background talking sorry.

I don't think there's anything wrong with wanting time estimates. And I honestly believe the engineers know approx. how long if they've done it before.

@amityweb:

A new concern I have is no one seems to care no time estimates are given - even if they are roughly known. It means the issue is unlikely to be addressed in future. I would have thought everyone would like to know time estimates so I am amazed people don't. The quality of service won't improve without the demand of customers. This could result in a slipping of service quality as competitors do improve such things. It's very important service quality is continuously improved. My engineering background talking sorry.
If linode tells me they are working on a problem, then I know from experience that they will get it fixed as quickly as possible. If I had to answer to a customer for every minute of downtime, then I would have long ago implemented redundancy/failover/whatever to ensure that a single linode failure would not impact my services. To me the quality of service is defined by how well they fix the problem, not how well they guessed how long it would take.

> If linode tells me they are working on a problem, then I know from experience that they will get it fixed as quickly as possible.
I dont disagree, I know they will do it as quickly as possible, with respect thats not the point. As quickly as possible could be 1 hour or 1 day, I would just like to know which.

No one will be able to make any argument or justification why its OK to not be given estimated timing. The end.

@amityweb:

No one will be able to make any argument or justification why its OK to not be given estimated timing. The end.

There's nothing surprising about the other customers not requiring or wanting estimates - they are experienced sysadmins who know what they are doing, know from experience that problems do not always have an ETA to a fix and that otherwise Linode works as fast as possible to solve those problems. We put redundant systems in place so that when there is unexpected downtime, we keep on going as if nothing happened.

I really don't want an update every 15 minutes telling me that there's still no ETA. It'd be more annoying than beneficial.

> I really don't want an update every 15 minutes telling me that there's still no ETA.
Thats the point of my post, thats what Linode say everytime I ask. What I wanted was for them to tell me it would be between 20 and 90 minutes as they stated above. They knew this but didn't tell me.

I want to make something clear that I am wondering if you all misunderstand… the estimated timing I think we should be given is based on previous experience and assuming it goes to plan. I am not asking for estimates based on something unknown going wrong. I would say most of the time things go to plan. If it never goes to plan then something else is seriously wrong somewhere.

This is a moot argument, you run a one box shop and are upset that things out side of your control impact your relationship with your clients. Others here know that A) estimates are a complete waste of time - making them, listening to them, expecting them to be anything close to accurate and B) if downtime is unacceptable, you need to implement system designs that include redundancy and failover. You're upset because you want a magic answer to everything and you either can't afford or can't design a redundant design. I'm guessing most people here can relate, most everyone starts out with the shoestring design - but you need to grow up fast if you're going to survive - for many many years your business will have to deal with the whims of many things you can't control - how you deal with those things will determine if your business survives, or like the vast majority, slides quietly below the waves.

TL:DR - either setup a redundant system, -or- explain to your clients that you don't have a redundant system therefore unexpected random downtime is a fact of life, -or- find a VPS host that provides the mythological estimates that you think Linode fails to provide. If you expect your business to survive and grow - the redundant system is the ONLY choice.

@vonskippy we have already established Linode knew the time estimates beforehand yet they did not tell me. The time estimates they knew were comparable to the time taken so it would have been very beneficial to know. I have also stated this is not about experiencing downtime I have accepted that happens.

The whole point of my post is its bad not to be told of this known time estimate. Why withhold it from me? I am an intelligent person, I know if something goes wrong the time may take longer than estimated. I would like to know times assuming it goes to plan.

If you personally do not want to know the time estimates then don't ask them for updates. But I would like know. We clearly run our businesses differently, so however you do it does not mean I should accept it. I have 12 Linodes, its not a one box shop… we're just a small company so do not have experienced sysadmins who know how to setup backup mail servers that are a complete duplicate of each other at all times in case one goes down. I would like to get there one day, if this is even possible, but at the moment I am not there, so all I ask is when a server does have an issue, to kindly let me know the known time estimate to bring it back. Like I said, they did know this, they just didn't tell me, and thats unacceptable.

> A) estimates are a complete waste of time - making them, listening to them, expecting them to be anything close to accurate
When I have been given estimates, by Linode or other providers, the majority of times they have been approximately met. Just because the very few times issues could occur that means times take a lot longer doesn't justify not being told an estimate if it all goes to plan. I accept there could be unknown things that could delay it, so if that happens let me know and I understand. But there is no justification to not give known timings if it is all goes to plan because of the small chance something may delay it. Every other industry use time estimates for planning, whether its new project development or fixing issues, there is nothing different in this industry to justify not giving estimates. All industries face unknowns. I take my car to the garage to have a MOT/Service, they say it will take 4 hours, and it mostly does, except that day they find the brake pads worn down and are out of stock which will add another 4 hours. Its a shame, but its acceptable this happens, but its still important to know that if it does go to plan then it will take 4 hours, because most of the time it does, and so my following planning based on this will mostly go as planned too and most of the time I can tell my wife I will be home for dinner at 6pm - if it goes to plan. Couldnt tell anyone anything with this issue.

Anyway, I think I have said what I wanted (more than planned sorry) and hope you understand where I am coming from. I dont want to keep going on about it (too late you'll say :)) so I'll leave it at that and will try harder to tease out some known timings from Linode in future (rare) issues like this. Thanks for everyones time and arguments! :)

One thing I learned being on the service provider end of things: never, ever give time estimates for unplanned events. If you miss them, they will come back to bite you. You are certainly a Customer Who Isn't A Vengeful Asshole, but there are enough jerks who tear into providers for giving estimates that proved optimistic that I won't even give estimates for planned network outages at home.

You will eventually learn this if you start giving time estimates to your customers ;-)

Good point. I am not vengeful and am understanding if things don't go to plan. This may be why I'm annoyed for not being told. Linode don't know that though so maybe they don't want to give estimates just in case. In which case I'm not sure how I could be treated any differently. Maybe next time I'll try to make it clear I just want an idea of timing based on it going to plan based on experience and I totally understand things might go wrong and I promise not to complain if they do. Maybe I could get some info then. Things usually go to plan in my experience so I don't see the harm in knowing. Live and learn, I'll try requesting it in a different way next time.

I want to make it clear that I am of course very pleased of the actual support and fix and am very happy with Linode. This post was not about that, it was only about the information updates in the support ticket. I've used a lot of providers and they're still the best and I love them to bits.