Paid Daily Back service Fails 6 days ! Sutdent fData LOSS!!

Quality post sir.

Also, I'm guessing user error. You might want to send in a ticket to see if they can retrieve your data.

I wrote a message for support some days ago because a snapshot I made was failing. It wouldnt complete when I clicked on snapshot.

I was nto aware the entire backup system was completely failing and would be for a week. There is absolutely no user failure. The backup service at linode should be advertised as discontinued or completely out of service very clearly.

: (

Hi Robert,

Please accept our sincere apologies for the issues you're experiencing. As advised in your previous support ticket (#488819), the backup service in the Fremont, CA is currently undergoing maintenance which is preventing backups for your Linode from being taken. Our backups team have been working diligently to restore functionality, but unfortunately some unforeseen circumstances has been delaying this process. Once the maintenance is complete and the Linode Backup system is back in an operational state, backups for your Linode will begin processing normally.

With that said, I have spoken to a Linode Backups administrator and I've been advised that we would be able to restore your last successful snapshot (taken: 2011-06-15 02:04:36) back to your Linode if you require access to it. Again, I'm truly very sorry for any inconvenience this may be causing you and we'll certainly do what we can to assist you.

Please let us know if you would like to have the 6-day old backup restored.

-Danny

@AntiNSA:

The backup service at linode should be advertised as discontinued or completely out of service very clearly.

The rest of the the data centres are fine so no, however something on status.linode.com for fremont would be good. I imagine if you request a refund for the failed time you'll get one.

While the failing backups in Fremont are indeed unfortunate, this ought to be a lesson to all of us to never trust our backups to a single entity. I would highly recommend performing your own backups in addition to the Linode backups. If you ever need to restore from your own backup, it's not going to be as seamless as restoring from one of the Linode backups, but at least you'll have your data.

Linode should really do something about Fremont. All these glitches and outages can't be good for Linode's reputation. Time to find another West Coast datacenter, guys!

@obs:

@AntiNSA:

The backup service at linode should be advertised as discontinued or completely out of service very clearly.

The rest of the the data centres are fine so no, however something on status.linode.com for fremont would be good. I imagine if you request a refund for the failed time you'll get one.

I think they still are billing me for it even . not time to check. killer amount of work due to data failure in linode..

Well thats life : ( .

so i guess my question is.. what caused your failure in the first place? Not the fremont issue, but the one on your linode that caused you to want your backups? Was that user error or..?

and was the last successful backup able to help you or not.

Likewise, have you been checking the status of your backups, or just assuming that everything is fine?

Because i know if i noticed that my backups were failing constatly, apart from logging a support issue, id be ensuring that i had my own backup somewhere incase i needed it..

@Internat:

Likewise, have you been checking the status of your backups, or just assuming that everything is fine?
Do you suggest everyone of us should login to linode control panel daily to check backup status?

@neo:

@Internat:

Likewise, have you been checking the status of your backups, or just assuming that everything is fine?
Do you suggest everyone of us should login to linode control panel daily to check backup status?

If the data or service that your linode is providing is that sensitive, critical, or otherwise important then the question should be:

why arent I checking my linode backups daily if that is the only backup I have when I know I should have offsite too because my stuff is to important to lose?

@Internat:

so i guess my question is.. what caused your failure in the first place? Not the fremont issue, but the one on your linode that caused you to want your backups? Was that user error or..?

Does it matter? I mean, thats what daily backups are for, right? I am developing things, and sometimes need to call on backups..

> and was the last successful backup able to help you or not.

6 days ago is an eternity > Likewise, have you been checking the status of your backups, or just assuming that everything is fine? I had to take a train from mainland china to hongkong, and habve a distand relative in the us open a bank account just to assume the backups were going to be "ok" > Because i know if i noticed that my backups were failing constatly, apart from logging a support issue, id be ensuring that i had my own backup somewhere incase i needed it..

I thought I was making sure when I paid for the service. Go figure.

@tonymallin:

@neo:

@Internat:

Likewise, have you been checking the status of your backups, or just assuming that everything is fine?
Do you suggest everyone of us should login to linode control panel daily to check backup status?

If the data or service that your linode is providing is that sensitive, critical, or otherwise important then the question should be:

why arent I checking my linode backups daily if that is the only backup I have when I know I should have offsite too because my stuff is to important to lose?

Thats insane. If the backup service is sensitive, and the reputationis of a reliable host, you are paying for that reliability and reputation. You are paying to not have to check everyday.

AntiNSA… As an aside… every post I've ever seen from you, you're flying off the handle about one thing or another (often being very sarcastic and antagonistic).

Obviously the backups shouldn't have failed, and I expect Linode will make it right (as they've always done in the past).

My guess is, you'll get a lot further with people if you tone down your language a bit. Right now you come across like some crazy person ranting and raving all the time. That's not always the best way to get your problems solved.

No, it's not insane. Any admin worth anything should be testing and checking their backups on a regular basis. If you're not, well…. This stuff tends to happen to those who are not testing their backups.

I've seen it dozens of times over the years as some small company thinks they have a backup solution in place. Some non-tech person has been shown how to change the tapes out daily and take them home. OOPS, server crashed. Their "IT" consultant comes in and tries to restore from the tapes, OOPS, they haven't been tested lately and are no good.

Even with the backups from Linode, it's not something I'd 100% rely on EVER. They are taking a snapshot of your system in time. If you catch some process like MySQL in the middle of doing something critical. So you have no idea what kind of state your database is in when that snapshot is taken. Sure it might be fine most of the time, but it seems the chances of it not being good and that being the backup that's needed is always pretty high….

Also, since these are snapshots of your entire system. For me, it's only really something I'd use (am using) for an emergency backup. In my experience, backups are more frequently accessed for file level restoration. In such case, Linode's snapshots a kind of a pain to do that.

I commented in another thread that Linode should consider adding alerting capabilities for backup status. If you don't want it, turn it off. If you do, then enter your email address and have it email you with the status of the last backup.

Right now I'm logging into the console daily, but at some point, I'd really rather not have to do that, just to see whether backups failed.

(And they failed again in Fremont last night. Sigh)

Thanks,

bruce

> (And they failed again in Fremont last night. Sigh)

My scheduled backup ran successfully in Fremont last night (this morning). Times are PST:

Started                  Finished               Duration                   Status
2011-06-23 01:10:51       2011-06-23 01:23:47    12 minutes, 56 seconds      successful

@tonymallin:

@neo:

@Internat:

Likewise, have you been checking the status of your backups, or just assuming that everything is fine?
Do you suggest everyone of us should login to linode control panel daily to check backup status?

If the data or service that your linode is providing is that sensitive, critical, or otherwise important then the question should be:

why arent I checking my linode backups daily if that is the only backup I have when I know I should have offsite too because my stuff is to important to lose?
So you are saying Linode service is so bad that we can not expect even the basic things to operate as promised, and should constantly manually recheck absolutely everything Linode service is supposed to provide? I know some people have problems with Linode, but overall I personally had a very positive experience, so I don't agree with your implication that Linode can not be trusted to do even the basic things right.

This backup fiasco dragged on for quite some time, and constitutes a serious failure. At the very least Linode should send email notifications when Linode backup fails.

What I'm saying is if the data or service that your linode is providing is that sensitive, critical, or otherwise important, and you have ONE method of backup, which you've never tried, and which may not be the best backup method for your situation, then why arent you checking your backups daily?

tl;dr: you need to lookup the best backup methods for your needs. In my experience, linode snapshots are best for short term backups of the OS disk and are rarely the most effective method of backing up actual customer or critical data that you may need to move to another system at short notice.

you're right, dont trust anyone backup method. That includes linode.

EDIT: added word

In any case, is this the proper forum for this post?

One user's problem belongs on a trouble ticket (or with his wallet - if he doesn't like it, walk away), not in Customer Testimonials.

This is just an under handed dig at Linode without any context or rebuttials.

I'm sure the positive testimonials out number the negative by several thousand to one.

Kudo's to Linode for not just deleting this thread as a troll.

@vonskippy:

Kudo's to Linode for not just deleting this thread as a troll.

Anyone even considering giving any credence to anything the OP of this thread posts should really read his history. troll is an understatement

@glg:

Anyone even considering giving any credence to anything the OP of this thread posts should really read his history. troll is an understatement
Nevertheless, the problem was real, many others experienced the exact same issue.

What drags these discussions on and on is the constant attempts of some posters to always "defend" linode no matter what actually happened, and blame users instead, in this case with ridiculous assertions like "you should have manually checked status of linode backups daily".

neo++!

Although it is irresponsible to solely rely on Linode's backup system – it's not meteor- or bankruptcy-proof -- and not to occasionally check that it's working.

Perhaps, but it really should notify you when it fails.

I'd like to share a recent experience I had with myhosting.com, on June 8th 2011 my VPS went down together with 30 other VPSs on this particular host node. The cause of the failure was according to their status blog multiple disk failures resulting in data corruption. It took three days of outage before they could migrate my VPS to another host node. I spent my three days trying to appease irate clients who could not access any of their websites. One even threaten to file a law suit! I had nightly website backups occurring and stored locally on the VPS including a paid backup which was supposedly a base backup for Open Vz containers. None of which helped in this unfortunate incident. I learned an invaluable humbling lesson have OFF SITE BACKUPS!

I have since then improve my backup procedure.

Linode VPS

Linode backups nightly

Websites backup locally Monday - Sat

Weekend backup locally Sunday

Full Website backups SCP to another server in VA - Sunday

Incremental backups SCP to same server in VA - Monday - Sat

Do a test restore at a different location.

I get an alert e-mail if any of the above website schedules fail, except for the ones created through Linode. I would agree that there should be some alert notification when Linode's backups fail.

I only use snapshots just before applying any huge updates or major setting changes.

I have to point out to be fair to myhosting.com they did issue me credit for two months. I now use my VPS account with myhosting.com solely for testing until my credit expires.

All of my production websites have been moved to Linode. Knock on wood, so far so good.

I do the same thing. I use virtualmin to do a full backup on Sunday and differentials Mon-Sat. They are then encrypted and copied to an SSHFS (Fuse) mount that is mounted to my backup server that is actually a VPS on its own at another much-cheaper provider.

The backup VPS is not one I would use on a daily basis as my production box (again, notice the cheap in my last sentence), but it doubles as not only a place to put my backups on but also bring online as my production server fairly quickly if the need arises.

I have no pity for somebody that does not create their own offsite backups. Lack of knowledge is no excuse… learn if you don't know how!

Linode should definitely notify when a backup has failed. Not building this feature into the backup system is in my view a major oversight especially if they allow failure to linger or cough "maintenance" to linger 6 days. 6 days fing days = we fucked up and dont know how to fix it yet, we dont consider backups important so we won't let you know 'you should be checking on us', strange given how Linode would certainly have a full port-mortem if any VPS nodes were down for 6 minutes.

However, ideally, one should still check that backup are being taken, and occasionally verify their backups - even if it costs 1 day of Linode time.

It reminds me of time when I worked in a small IT department some 15 years ago. We had a IT manager, 2 competent IT sysadmins, 1 developer for Excel/VBA etc. We had one of component IT sysadmin change the backup tapes every day at 4.30pm, and walk to other side of campus and put them in a safe, occasionally they would be verified. Now I think of this situation as Linode customer = IT manager, IT backup boy = Linode backup service. Whilst it is feasible that the IT backup boy might use those 20 minutes to go smoke some pot in some hidden area of campus and do the backups once every dozen times instead (and not mention it), it would be a major offence, the IT manager would be pissed and surprised and all hell would break lose when the offence came to light. Sure the IT manager hold the ultimate responsibility (and it rests on his head lalala) and verification by the other sysadmin is no doubt a good idea, but we dont really expect a competent sysadmin to using the backup time to go smoke pot and not tell us, we trusted this guy at least in his professional capacity. The pot smoking IT sysadmin needs to come forward, needs to volunteer that he has not been making backups, needs to admit that's not just a controllable hobby (maintenance) but that's he mentally messed up and needs some time off to get back in order, in the meantime the IT manager needs to give the backup responsibility to someone. He should probably be disciplined as well, possibly a salary cut.