Accounts hacked via customer support interface: any update?
For more details please head to this pastebinhackernews thread
I'm sure Linode' staff is busy sorting this out and I have all confidence until proven wrong but a little timely update to your customers would be appreciated.
Thanks
233 Replies
> Pretty much, if you're a @Linode customer? Your system has almost definitely been hacked and rooted. Because they had a global superuser.
Is this true? Linode says otherwise, however if they admit to something of this extent, they would fear losing their customer base.
By the time anyone notices the reboot, it's done, and the only evidence consists of a Tor exit node IP and a Bitcoin address. Heck of a heist, that's for sure.
Linode will probably post a full postmortem report in a few day's time, not in that obscure status site, but in their official blog this time. Security breaches can happen to anyone. What sets responsible companies apart from the rest of the herd is how they handle emergencies like this. I trust that Linode will respond professionally.
About 3 years ago, a budget OpenVZ virtual hosting company with thousands of customers got completely destroyed, all data lost, allegedly because of an unpatched bug in the then-popular HyperVM customer portal. The Indian guy who sold HyperVM committed suicide the next day. What followed was one hell of a mess. But Linode ain't like that, is it?
Nope.
They just told me they have nothing else to report at this time.
So I will be moving off of Linode and telling everyone I know to do the same. The complete lack of transparency is unacceptable.
@taligent:
Nope.
They just told me they have nothing else to report at this time.
People who have nothing else to report "at this time" often have something new to report after a few days.
That is the exact quote. So I would not be holding your breath.
@taligent:
So I will be moving off of Linode and telling everyone I know to do the same. The complete lack of transparency is unacceptable.
So where will Aunt Betty and your 3 D&D pals be moving to?
Knee jerk reactions with ZERO evidence that there is some big cover up is childish to the extreme.
I'm sure you and your tinfoil hat will have a great time moving to somewhere 100% safe.
Coincidence? Probably not.
What more can we ask?
At least they have an audit trail!
@vonskippy:
Knee jerk reactions with ZERO evidence that there is some big cover up is childish to the extreme.
I'm sure you and your tinfoil hat will have a great time moving to somewhere 100% safe.
Eh ? At what point did I ever suggest there was a cover up.
I just wish that more information was provided much, much earlier. The same behaviour was exhibited when there was a power outage at Fremont.
This is in no way a reflection of the engineers/admins at Linode who are always quick to respond to questions and supremely helpful.
It's just unacceptable that I should have to read about this on Reddit before hearing from Linode.
@artagesw:
What bothers me most about this is, assuming the perpetrator was not a Linode employee, Linode's backend customer support interface apparently is accessible over the Internet when it should be locked down and accessible only from designated internal hosts. That's a huge backdoor to every Linode just begging to be opened.
What's the point of saying things like "assuming" and "apparently" and then making a conclusion.
I believe linode aren't transparent enough, they don't keep customers updated very well.
Fact is that I heard about the security breach on Slashdot and then couldn't find anything on Linode's site until browsing the forums – I would expect an e-mail to all of their customers whenever a security breach happens, a proper explanation of why it could happen, and how they are altering their system to prevent any future incidents.
Since they didn't address the last two issues, I would advise any business with valuable data to seriously reconsider Linode and get in touch with them. It's nice that they contacted all those affected, but not enough when security comes into play.
Since the problem was on Linode's side, it's funny that they didn't even tell if they will compensate for the damages.
from the pastebin logs this took about 6 hours to resolve
the question you have to ask yourself is how would every other hosting company/VPS deal with that situation? would it be fixed in that time? would you get a response from the senior management team at that company?
"Subscriber further acknowledges that Linode.com's liability for its own negligence may not in any event exceed an amount equivalent to charges payable by subscriber for services during the period damages occurred."
I guess Linode should be a no-go for any serious business anyway.
@taligent:
Linode will probably post a full postmortem report in a few day's time
Nope.
They just told me they have nothing else to report at this time.
So I will be moving off of Linode and telling everyone I know to do the same. The complete lack of transparency is unacceptable.
Oh look at what the cat dragged in.
There will be a follow up post for sure. They don't have anything else to report for now, currently, at this moment, this very second. Is that clear enough for you?
Given this companies history I'm more than happy to give them the time they need to follow up and make any changes to policy needed. Finally, anyone taking advice from you is clearly in way over their head. So you taking them with you isn't really saying much.
@bcoker:
Oh look at what the cat dragged in.
Seriously ? Is this really necessary or appropriate ?
@bcoker:
There will be a follow up post for sure. They don't have anything else to report for now, currently, at this moment, this very second. Is that clear enough for you?
Care to explain how you know this ? I am just basing my actions from what Linode has told me directly. If you know something I don't then I am sure it would be useful for everyone here.
I don't think I am being unreasonable here. A rogue third party with the ability to instantly get root access to all my Linode servers is a serious issue, no ?
@taligent:
Seriously ? Is this really necessary or appropriate ?
I certainly do. Your knee jerk reaction show's a lack of knowledge of the situation and the industry. Your threat to take customers with you just reinforces that.
Sure, I'll explain how I know. I know because I've been in the business for 17 years. From Floor grunt to Boardroom. If that's not good enough for you then so be it.
They will with 100% certainty create a follow post/article because they know people like me demand to know the details of their after action report. Not to mention the harm it would cause them within the industry as their name was blasted for not doing so. They may lie right thru their teeth about the findings but they will do so either way.
Your concern is not unreasonable. In any way. Yes, it's very unsettling that someone had access to your server that shouldn't have. The reasons why you are stating you are concerned are. Based off ignorance of the situation and knee jerking. It's not an uncommon reaction by some under such circumstances but doesn't make it reasonable or logical.
I'm no linode fanboy here and don't confuse my reasonability and logical approach for weakness of some kind. I'm just not idiot enough to draw my shotgun and start blasting people because I don't know what the whole deal is yet.
Again, if that's not good enough for you then so be it.
I recently moved to Linode and it's a bit disturbing to see it compromised in such a way. They will have to improve their security after this incident, perhaps introduce additional security features on Linode manager.
Stuff like this happens to every provider now and then unfortunately, and the only thing that really separates the providers in this area is how they deal with it afterwards. At this stage, it seems Linode has done everything right. We know what was done, who was affected, etc, and we knew about it the same day that it happened.
What I want to know now is what steps Linode will take for ensuring this exact scenario will not happen again. It seems official login credentials were used to perform this attack which means that either a support-level employee was careless, or even part of the attack. A possible way to resolve this is adding a higher level person to sign off on stuff like changing root passwords, it would prevent a similar thing from happening again.
But I don't need to know this right this instant.
@taligent:
Seriously ? Is this really necessary or appropriate ?
I think it is entirely appropriate - and in your case, necessary.
James
@compizjoe:
I does sound like a simple case of slightly bad wording. They've informed people about exactly what has happened, and they did so quickly.
No they didn't, they put up a posting on status.linode.com. If there's a serious security breach, they should reach out to me, I shouldn't have to check that site every day to make sure somebody didn't hack in again.
@OverlordQ:
@compizjoe:I does sound like a simple case of slightly bad wording. They've informed people about exactly what has happened, and they did so quickly.
No they didn't, they put up a posting on status.linode.com. If there's a serious security breach, they should reach out to me, I shouldn't have to check that site every day to make sure somebody didn't hack in again.
According to their investigation, you were not affected by the breach; they did reach out to affected customers. As they say in that post.
Would you also expect to be notified when Billy Joe Bob's linode suffers from a host disk failure, even though you don't know him and don't have any linodes on his host?
@Guspaz:
@OverlordQ:
@compizjoe:I does sound like a simple case of slightly bad wording. They've informed people about exactly what has happened, and they did so quickly.
No they didn't, they put up a posting on status.linode.com. If there's a serious security breach, they should reach out to me, I shouldn't have to check that site every day to make sure somebody didn't hack in again.
According to their investigation, you were not affected by the breach; they did reach out to affected customers. As they say in that post.
Would you also expect to be notified when Billy Joe Bob's linode suffers from a host disk failure, even though you don't know him and don't have any linodes on his host?
First off, all customers were affected, but supposedly only 8 were tampered with further.
Second, no I wouldn't expect to be notifed of that because it doesn't affect me. Critical management infrastructure being broken into, does.
@OverlordQ:
@Guspaz:
@OverlordQ:No they didn't, they put up a posting on status.linode.com. If there's a serious security breach, they should reach out to me, I shouldn't have to check that site every day to make sure somebody didn't hack in again.
According to their investigation, you were not affected by the breach; they did reach out to affected customers. As they say in that post.
Would you also expect to be notified when Billy Joe Bob's linode suffers from a host disk failure, even though you don't know him and don't have any linodes on his host?
No, because that doesn't affect me. The failure to protect critical management infrastructure does.
Exactly, spot on. I need full details on whether this has anything to do with Linode's control panel. I don't want a Vaserv scenario.
I really love linode but I don't have a better place to go.
But what I really need is the removal of the root password change from the interface and api's.
If I lose it or forget it… Shame on me. Nobody should be able to change it…period.
We need to be sure that nobody can change that not even linode.
If I can do it from an interface then any will (authorize/unauthorized).
@pic.micro23:
But what I really need is the removal of the root password change from the interface and api's.
If I lose it or forget it… Shame on me. Nobody should be able to change it…period.
I agree with this 100%.
@zunzun:
I think it is entirely appropriate - and in your case, necessary.
Really ? Wow. Personal attacks for simply being critical that loyal Linode customers should have been better notified about a major security incident that directly affects them.
This is officially the worst forum I've been to in years.
Just connecting a computer to the internet is a security risk.
If succesful hacking happens to the bods as NASA, then why do some people assume that Linode, or any other hoster is immune?
@artagesw:
@pic.micro23:But what I really need is the removal of the root password change from the interface and api's.
If I lose it or forget it… Shame on me. Nobody should be able to change it…period.
I agree with this 100%.
You understand that this is just a "nice" feature right? if someone has access to your linode manager account, they can just reboot the node into single user mode, open lish, reset the password and reboot.
Or boot into finnix and do the same thing.
Yes this feature makes it convenient and easier, but its not a security issue.
@taligent:
@zunzun:I think it is entirely appropriate - and in your case, necessary.
Really ? Wow. Personal attacks for simply being critical that loyal Linode customers should have been better notified about a major security incident that directly affects them.
This is officially the worst forum I've been to in years.
+1
Being rash is one thing, but attacking another person personally shouldn't be accepted (and definitely not endorsed!) by the Linode community. Making wild and potentially hurtful speculations about one's personal life or social status is unacceptable behavior (more so if all he did was demand being informed about security breaches that could easily have compromised his own Linodes.)
@jk4736:
@taligent:
@zunzun:I think it is entirely appropriate - and in your case, necessary.
Really ? Wow. Personal attacks for simply being critical that loyal Linode customers should have been better notified about a major security incident that directly affects them.
This is officially the worst forum I've been to in years.
+1
Being rash is one thing, but attacking another person personally shouldn't be accepted (and definitely not endorsed!) by the Linode community. Making wild and potentially hurtful speculations about one's personal life or social status is unacceptable behavior (more so if all he did was demand being informed about security breaches that could easily have compromised his own Linodes.)
Think of them as rabid apple fanboys. You attack the product, you get attacked.
In any case, what happened to Linode is SEVERE. I agree an email should have been sent out to ALL clients notifying us of the breach instead of those affected.
Why? It DOES affect ALL of us, not only the 8 that was breached. They had master root access, who knows what they could've done to the rest they didn't have time to dig through?
They had "support representative" access.
If your linode hasn't been rebooted, if your linode's root password hasn't been changed, you haven't been affected.
@rsk:
They didn't have root access.
They had "support representative" access.
If your linode hasn't been rebooted, if your linode's root password hasn't been changed, you haven't been affected.
Hence, if they emailed us explaining in more detail what happened and what did not happen. We would be more informed and less likely to be upset.
I would much prefer to hear that my provider has had a security breach directly by email from the provider themselves. Or who knows, maybe we'll get that email after the audit has been done.
One also has to consider the unnecessary panic such an announcement might cause among customers.
I'm sure they will do as well as they have in the past.
fos
@rsk:
They didn't have root access.
They had "support representative" access.
If your linode hasn't been rebooted, if your linode's root password hasn't been changed, you haven't been affected.
I love Linode. I really do. I've been a personal customer for many years. I no longer have my own Linode, but still manage about a dozen or so for clients.
I find the above statement confusing. How does 'support representative' access allow access to 8 unrelated accounts (ie - different account holders, different accounts, probably different datacenters), yet not allow access to all other accounts?
To me, and probably to others, if the intruder(s) can attack 8 independent Linodes, then they can attack them all.
Perhaps the Linode team can clarify this?
Also - rumor has it that it was an inside job. Was this the case?
I agree with the sentiment that more information on exactly what happened is needed.
@rainkid:
I find the above statement confusing. How does 'support representative' access allow access to 8 unrelated accounts (ie - different account holders, different accounts, probably different datacenters), yet not allow access to all other accounts?
To me, and probably to others, if the intruder(s) can attack 8 independent Linodes, then they can attack them all.
Perhaps the Linode team can clarify this?
Also - rumor has it that it was an inside job. Was this the case?
I agree with the sentiment that more information on exactly what happened is needed.
As far as I can tell from the information given, normal support level login credentials were used. In other words, no exploit of any kind (Except the human kind) was used, which means they are able to see exactly who was affected through the logs. Yes, the person could in theory have attacked all the nodes, but he or she didn't. There's nothing that needs clarifying about this part specifically, there's no risk of repeat with the same credentials, and nobody else was affected or can be affected in the future as a result of this specific hack.
While a more comprehensive report certainly will be interesting to read, there's no more immediate information that Linode needs to give, the question you asked has already been answered by the official information given. (Or an answer is possible to infer easily).
As far as inside job goes, that would be interesting to know but ultimately doesn't matter that much right now. It's the kind of thing that will be interesting to read in a more comprehensive report of the incident.
@OverlordQ:
Second, no I wouldn't expect to be notifed of that because it doesn't affect me. Critical management infrastructure being broken into, does.
It technically wasn't broken into the same sense as someone hacking it. Someone had login credentials, logged in, did stuff. Linode knows exactly who was affected, and there's no risk at all of this affecting you if you haven't received a notification already. There simply isn't any need for an immediate notification to every customer when we know for sure that only a very specific set was affected and nobody else will be affected in the future.
There is a necessity to give some information to the general public about what had happened, sure, but they did that. Emails should only be used if people are required to take some sort of action. If you want to get status updates in general quickly, subscribe to the feed on status.linode.com. You can probably even get that as an email through some service.
Essentially, all we are told is 'someone had access, and did bad things. we removed said access.'
Not very informative.
@compizjoe:
@OverlordQ:Second, no I wouldn't expect to be notifed of that because it doesn't affect me. Critical management infrastructure being broken into, does.
It technically wasn't broken into the same sense as someone hacking it. Someone had login credentials, logged in, did stuff.
Not to add oil to the fire but as far as I can see, there is nothing in the status update to give the impression that it "wasn't broken into the same sense as someone hacking it."
It could easily have been and the status update would still make sense. I think Linode is being deliberately vague at this point so as not to commit either way.
I do know a bit about this world that we are discussing, from both sides of the coin
@rainkid:
In that case - how did the attacker(s) gain this level of access? How do we know that they no longer have this level of access? If those credentials are no longer valid, how do we know that the attacker cannot acquire new credentials and wreak more havoc?
Essentially, all we are told is 'someone had access, and did bad things. we removed said access.'
Not very informative.
I suspect finding out exactly how the attackers stumbled upon those credentials will take some more research. But it not reasonable to assume every single credential is also vulnerable. If someone gains unauthorized access to my system using credentials one of my users had written down somewhere, I would, as a system administrator, not then assume the login credentials of every account had become vulnerable. If the attackers did have more extensive access than a simple login credential, then it seems foolish to go through a process where their actions are immediately obvious and logged when they could simply just do whatever they wanted directly.
So either they're so smart they've been able to gain some kind of superprivileged access to the system, yet dumb enough to not use it, or this is simply a case of one login credential getting used by the wrong people. My money would be on the latter.
Why does everyone go out of their way to construct a movie plot threat out of this?
@skn:
Not to add oil to the fire but as far as I can see, there is nothing in the status update to give the impression that it "wasn't broken into the same sense as someone hacking it."
It could easily have been and the status update would still make sense. I think Linode is being deliberately vague at this point so as not to commit either way.
I do know a bit about this world that we are discussing, from both sides of the coin
:twisted:
Yes, the status update was a bit vague, I completely agree. After all, it was written only some hours after the event had occured, I suspect it's more a case of limited knowledge at the time rather than a conspiracy to keep people in the dark.
@rainkid:
I find the above statement confusing. How does 'support representative' access allow access to 8 unrelated accounts (ie - different account holders, different accounts, probably different datacenters), yet not allow access to all other accounts?
To me, and probably to others, if the intruder(s) can attack 8 independent Linodes, then they can attack them all.
From my understanding, they got a login/password belonging to one of the Linode support reps - the first level of people who receive your support tickets. Logged in as him, and used the "reset root password for node" option for eight nodes. Everything was logged in audit trail, so Linode knows what happened.
I sure hope I'm not mistaken about it, and we should demand a more detailed report - but in a reasonable time from now. Give'em at least a week to respond before starting a riot.
Need a new best practices manual.
> We believe an unauthorized party gained access, for a period of time
Why does this sound familiar?
Two hosting companies within a day of each other…
How?
It's very likely that the attacker was from the inside, or had MUCH more access than we're told. How else would they know which (independent) accounts to target?
Again, not knocking Linode (they're great and I've told them this many times in the past), but something is amiss here.
@rainkid:
The attacker somehow got the login credentials of a Support Rep, yet, knew EXACTLY which eight accounts to target.
How?
It's very likely that the attacker was from the inside, or had MUCH more access than we're told. How else would they know which (independent) accounts to target?
Again, not knocking Linode (they're great and I've told them this many times in the past), but something is amiss here.
Agreed. Linode may want to give us some more information soon to calm our wild imaginations…
@nehalem:
Agreed. Linode may want to give us some more information soon to calm our wild imaginations…
I'm pretty sure there's nothing more they can really say that would quiet the more active imaginations.
See "birthers", for reference.
@nehalem:
Agreed. Linode may want to give us some more information soon to calm our wild imaginations…
My neighbors friend has an uncle that knew someone that read on Hacker News that Linode will be sending out partial rolls of generic tinfoil in the next billing cycle, and will include not three, but four unique ways to fold them into tinfoil hats guaranteed to protect you from the outer space Nargles and the local FBI.
@rainkid:
The attacker somehow got the login credentials of a Support Rep, yet, knew EXACTLY which eight accounts to target.
All of the victims were Bitcoin dealers. Machines running Bitcoin software are readily identifiable through a port scan or through transaction records with other Bitcoin machines. Once you know which IP addresses to target, it would be trivially easy for someone who has the credentials of a support rep to figure out exactly which accounts to break into.
@hybinet:
All of the victims were Bitcoin dealers. Machines running Bitcoin software are readily identifiable through a port scan or through transaction records with other Bitcoin machines.
See what you did? Injected actual facts into the tinfoil hat party, and totally killed it!
It seems to me that all the responses from people saying "they need more time" have pretty much lost their weight.
I absolutely love Linode but the way they just ignore the forum posts about this, as if hoping it just goes away is VERY bothersome.
I hope others join me in continuing to pursue this subject.
I want to keep loving Linode and being a faithful customer for years to come but for that to happen I have to be able to trust them and for that to happen, they have to be open, good or bad, just like we all learned while growing up.
It's their MO.
@chesty:
Linode is extremely secretive. It always has been. I lost a bet that they acknowledged the break in, in the first place, i didn't think they would. I'm very confident you won't hear anything else from them about the subject.
It's their MO.
That is disapointing to hear, for two reasons.
The first reason is obvious. I want information and they aren't sharing.
The second reason is that, if what you say is true, it also makes them liars.
In the statement they did release, they open by saying…
> Ensuring the security of our platform is our top priority. We maintain a strong security policy and aim to communicate openly should it ever be compromised. Thus, we are posting to describe a recent incident affecting the Linode Manager.
If you have a policy of openness, remain open about these things, otherwise just change the policy.
If you want to remain quiet in these situations, at least come out and say so. It would be much better to just tell us "Something happened, those affected have been notified, we will discuss nothing further on the matter.". But that isn't what they did, they made a public announcement basically claiming that they have an ongoing policy of openness.
I hope at some point someone high up just finally comes out and openly discusses this OR just tell us that they have no intentions of doing so. Ignoring something to make it go away is not the proper course of action.
12
Is there anything in specific beyond this that ought to be disclosed, and doesn't fall into the realm of Things That Would Jeopardize FBI Investigations or Things That Would Violate Privacy Policies And Laws?
I, too, would like to know more, but I don't know what practical use that knowledge would be. If Linode had more of my personal information, I'd probably demand credit reporting service or something, but they don't, and I already have it
@worldviewpr:
would any of this explain why I can't access either webmin or ftp?
nope..
@hoopycat:
A large part of it might be that there's nothing more to say: a customer service password was brute-forced, someone used it to "recover" access to a handful of accounts
The story on the register claims that Linode network equipment got p0wned. My guess is that network sniffers were then used to grab a customer support password or maybe an authentication cookie.
I don't like guessing. People that trust me have minor stuff on Linode because I recommended it. I also have some personal stuff on Linode. I can't estimate risk on guesswork alone.
So what do I do if I can't trust cloud stuff, virtual machines, or even dedicated machines that are plugged into someone else's remote management setup? Go to all the expense and hassle of setting up a server myself and getting it colocated?
@Typo:
I hope at some point someone high up just finally comes out and openly discusses this OR just tell us that they have no intentions of doing so.
I opened a support ticket to asked about this. I was very politely told by Caker himself that they would not be giving out any more information.
If anyone else wants to open a support ticket to ask the same question feel free. Maybe they will cave if they get 100 of these things.
@sednet:
I opened a support ticket to asked about this. I was very politely told by Caker himself that they would not be giving out any more information.
If anyone else wants to open a support ticket to ask the same question feel free. Maybe they will cave if they get 100 of these things.
That just sucks. How is it that more people aren't demanding information, especially after it was promised by caker himself on page 10 of this post in reply to a request for more transparency.
I just cannot believe the policy is to just ignore it after that kind of public promise to give information.
Here is the conversation from the other post….
@caker:
@scaredpoet:Between the furor over the Bitcoin incident and the beating linode took over its lack of cooperation in the lowendbox/lowendtalk DDoS incident, I for one am re-evaluating whether I want my VPSes here or whether it's time to move on. Taken in total with my own experiences with Linode support in the past, and the attitude towards things like IPv6 migration, I think it's fair to say that Linode is quickly earning a reputation for being not as customer friendly as they were once thought to be.
Very much appreciate your comments.Since last week, we've been completely consumed with evaluating, discussing, debating, planning, etc, ways in which we can do better. This was a learning experience for us and Linode will only improve because of it. Hoping to have an announcement soon covering the results of these efforts.
With regards to the lowendbox thing - we handled it the same way we handle all network attacks. The forum post from those guys had ZERO effect on how it was handled. A threat of a DDoS never provokes preemptive action from us, unless the customer requests it. We left the forum post there in order to BE more transparent, if that makes sense…
Not sure what you mean regarding IPv6. What attitude? We've worked hard to make native IPv6 available to you guys, which it is now in all six of our facilities.
Thanks,
-Chris
@Typo:
That just sucks. How is it that more people aren't demanding information, especially after it was promised by caker himself on page 10 of this post in reply to a request for more transparency.
I just cannot believe the policy is to just ignore it after that kind of public promise to give information.
Here is the conversation from the other post….
@caker:
@scaredpoet:Between the furor over the Bitcoin incident and the beating linode took over its lack of cooperation in the lowendbox/lowendtalk DDoS incident, I for one am re-evaluating whether I want my VPSes here or whether it's time to move on. Taken in total with my own experiences with Linode support in the past, and the attitude towards things like IPv6 migration, I think it's fair to say that Linode is quickly earning a reputation for being not as customer friendly as they were once thought to be.
Very much appreciate your comments.Since last week, we've been completely consumed with evaluating, discussing, debating, planning, etc, ways in which we can do better. This was a learning experience for us and Linode will only improve because of it. Hoping to have an announcement soon covering the results of these efforts.
The following blog posts seem to cover "the results of these efforts":
@retrograde inversion:
The following blog posts seem to cover "the results of these efforts":
http://blog.linode.com/2012/04/05/event … rotection/">http://blog.linode.com/2012/04/05/events-rss-feed-emails-and-profile-protection/
http://blog.linode.com/2012/04/05/linod … rotection/">http://blog.linode.com/2012/04/05/linode-manager-brute-force-protection/
For one, no post I have seen has been made saying for sure what methods were used to gain access.
I am not going to just assume that a couple blog entries made like a month later about new features related to some of the most basic security features available to protect sensitive areas of a website are a response to what happened nor will I just assume that those two posts cleared everything up because I still don't even know the extent of what happened or if this is enough to make sure it cannot happen again.
I'm not sure how you came to the conclusion that those posts were anything even close to the promised announcement covering the results of the efforts they took to make sure this cannot happen. It wasn't even mentioned unless I missed it.
@Typo:
For one, no post I have seen has been made saying for sure what methods were used to gain access.
I'm not replying in defense of Linode, this post would be poor representation, but I can't help but to reply. What are you expecting them to post? A blow-by-blow or step guide to how the individual or group was able to accomplish the break in? Really? That's the last thing you should want them to post. Sure, I have an idle curiosity as well, but what it all boils down to is whether or not I have confidence in the fact it won't happen in the future. They've made a series of changes recently, presumably to prevent this from occurring again, its up to you to decide.
I know this falls in to the category of blaming the victim, but as far as the Bitcoin concept and operators go… 1) Maybe this is another example of why its not such a good idea. 2) If you're going to store sensitive data that is accesible via the Internet, you darn well better make sure its properly secured - and yes, that includes if someone has access to your Linode Manager account.
I don't expect every single little detail but the basics of what happened and how they have ensured it won't happen again would be fantastic.
It is not an unusual request, especially considering the nature of the situation. This company uses a completely proprietary manager which can, if exploited, get around the other security measures which may be in place to protect our nodes. Some of the users here have very sensitive data and/or clients with sensitive data and everyone in that picture deserves peace of mind given back after this type of failure in security.
I want to make it clear that I love linode. I loved this place even before I was a client, I think the service and setup just rocks. I would just like to see a little more openness when a mistake is made.
I think even a post saying something along the lines of "We have finished investigating the matter and have ensured that this cannot happen any more, sorry again, here's a free beer".
@Typo:
It is not an unusual request, especially considering the nature of the situation.
Its not unusual, but you also have to have a realistic expectation and beating it to death does not help. They have been responding, you just don't seem satisfied.
@Typo:
This company uses a completely proprietary manager which can, if exploited, get around the other security measures which may be in place to protect our nodes.
What does proprietary have to do with anything here? However they gained access, once you have access through Linode's proprietary system or your favorite open source Xen manager - you have access.
@Typo:
Some of the users here have very sensitive data and/or clients with sensitive data and everyone in that picture deserves peace of mind given back after this type of failure in security.
Then those users (and I say that like that because you didn't include yourself in your own statement) should be implementing multiple measures to ensure that data remains secure. If you give me access to your Linode Manager account and you're worried about me accessing sensitive data rather than just deleting data, you are doing something wrong.
Security is a matter of multiple layers, and none of them are absolutely effective. You're only real hope is to make it more painful than the gain and/or slow them down enough until more direct measures can be implemented. I imagine this is why they implemented the e-mail alerts. Though, I'm not sure why it doesn't send an e-mail alert when you change the alert setting from enabled to disabled.
@AVonGauss:
What does proprietary have to do with anything here? However they gained access, once you have access through Linode's proprietary system or your favorite open source Xen manager - you have access.
It has a lot to do with it, with "my favorite open source" app, everyone can see the code, if its unsafe, its usually discovered and reported and fixed. In this case, we just have to trust them.
@AVonGauss:
Its not unusual, but you also have to have a realistic expectation and beating it to death does not help. They have been responding, you just don't seem satisfied.
I don't feel I've beaten anything to death. I voiced a concern in reply to a post (in 2 different threads granted) and have responded to replies to those posts. Also, nobody with any authority has responded to any of the posts regarding this, nor has any announcement relating to it been released since we were told to expect one. I fail to see where "they have been responding" unless your referring to the recent security enhancements which again would just be assumption.
I just personally think that this long after the incident, the situation should be resolved and the announcement should have been made.
I want to say again. I love this place and think its a great service and I am in no way trying to start an argument or troll.
@Typo:
@AVonGauss:What does proprietary have to do with anything here? However they gained access, once you have access through Linode's proprietary system or your favorite open source Xen manager - you have access.
It has a lot to do with it, with "my favorite open source" app, everyone can see the code, if its unsafe, its usually discovered and reported and fixed. In this case, we just have to trust them.
@AVonGauss:Its not unusual, but you also have to have a realistic expectation and beating it to death does not help. They have been responding, you just don't seem satisfied.
I don't feel I've beaten anything to death. I voiced a concern in reply to a post (in 2 different threads granted) and have responded to replies to those posts. Also, nobody with any authority has responded to any of the posts regarding this, nor has any announcement relating to it been released since we were told to expect one. I fail to see where "they have been responding" unless your referring to the recent security enhancements which again would just be assumption.I just personally think that this long after the incident, the situation should be resolved and the announcement should have been made.
I want to say again. I love this place and think its a great service and I am in no way trying to start an argument or troll.
If Linode has chosen to investigate with the authorities then they may be legally obliged not to respond.
@tonymallin:
If Linode has chosen to investigate with the authorities then they may be legally obliged not to respond.
Except they could say "Due to ongoing investigations, we're unable to comment at this time."
Instead, they just ignore all the chatter.
My guess, people will start voting on their "opaque transparency" with their feet (or should that be wallet?).
They still list their "Marketing Guru" as open on their "We're Hiring" page, so no big surprise that there isn't any real change in how they handle stuff like this.
@vonskippy:
Instead, they just ignore all the chatter.
It doesn't matter where you draw the line here, someone is going to be unhappy.
Me? I'm happy.
@ericholtman:
@vonskippy:Instead, they just ignore all the chatter.
It doesn't matter where you draw the line here, someone is going to be unhappy.
Me? I'm happy.
I'm not really happy. It appears linode lost a customer support interface account that lets whoever uses it change the root password on any linode. We got lucky that the attacker appeared to be one guy with an interest in BitCoins. It could easily have been a hundred plus script kiddie hackers who could have prepared tools to loot these machines in advance. With one big server outside they could have copied off everything that looked remotely interesting from every Linode and sorted though it later.
@Mr Nod:
Do these posts actually get read by the Linode hierarchy though? Otherwise this is just flogging a dead horse…
Every post on these forums is read (or at least skimmed) by Linode (don't know if Caker has the time to read them all himself).
@Mr Nod:
Do these posts actually get read by the Linode hierarchy though? Otherwise this is just flogging a dead horse…
Unless Caker has had a personality transplant in the last few years he reads and cares about what people say here.
I wish I knew what he really thought about this. He must know that Linux hobbyists are an awkward bunch and won't just forget about something as serious as this.
> We maintain a strong security policy and aim to communicate openly should it ever be compromised.
For me and some of the others (I'm assuming), the little official information we got could in no way be construed as Linode "communicating openly" about what happened.
@sednet:
Unless Caker has had a personality transplant in the last few years he reads and cares about what people say here.
Rackspace bought his personality to use as the basis for their Fanatical Support. Now he's a grumpy BOFH.:(
@graq:
A certain competitor has been emailing it's customers explaining a security issue that existed and has now been fixed. Perhaps Linode are experiencing the same issue, but haven't found the fix yet.
I kind of assumed it was something in the manager but like every other idea we all throw out its just guess work and assumption because nobody who knows is talking.
How about more details just in case your hypothesis proves true?
@Typo:
How about more details just in case your hypothesis proves true?
Hey…. I had a dream last might where aliens from Zabron 9 broke in and stole some accounts.
Should caker sign on and deny this too?
@ericholtman:
@Typo:How about more details just in case your hypothesis proves true?
Hey…. I had a dream last might where aliens from Zabron 9 broke in and stole some accounts.
Should caker sign on and deny this too?
You don't need to be silly. You indicated that you are happy with the little information you got. We get it. Now, there are others who are not happy with the little information they got, and I think reasonably so.
Basically you're not helping.
@nehalem:
You don't need to be silly.
Some other competitor reports some other security breach on some other platform, and someone here wants a response to that, and I'm being silly?
@ericholtman:
@nehalem:You don't need to be silly.
Some other competitor reports some other security breach on some other platform, and someone here wants a response to that, and I'm being silly?
I asked for more information since the info he gave was seriously lacking. If he was honestly curious if the security issues were at all related, which I already sort of pointed out that I don't think it is, then he will need to provide more information.
We have enough lack of information in this thread to go around, I was just trying to lessen it a bit.
You are obviously posting simply to start trouble while we have an honest desire to get answers that were promised us and are directly related to the safety of our vps's.
Yes, YOU are being silly.
What else is needed? IP/name/photo of the attacker? Source code of the manager app? Detailed access logs? Names of the victim node owners? Firstborn babies?
@Typo:
You are obviously posting simply to start trouble while we have an honest desire to get answers that were promised us and are directly related to the safety of our vps's.
I am not posting "just to start trouble".
I am posting to say that I do think we've gotten answers. I think we've gotten all the answers we're going to get. And I am satisfied with those answers.
Am I not allowed to hold those opinions?
> What else is needed? IP/name/photo of the attacker? Source code of the manager app? Detailed access logs? Names of the victim node owners? Firstborn babies?
For me, I want to know how the attacker accessed the web-based Linode customer service portal. Was it brute-force? Was it phishing? Was it a vulnerability in the portal itself? Was it an inside job? How did the attacker target the bitcoin people so quickly? I would like concrete answers to those questions without the speculation.
> I am not posting "just to start trouble".
To me, it seems you are. You already posted:
> It doesn't matter where you draw the line here, someone is going to be unhappy.
Me? I'm happy.
And we get it. You are happy with the response you have gotten. Good for you. And you expressed that you are satisfied. If everytime someone who is not happy posts somethings, and you reply by saying you are happy, then yes, it does seem you are trying to start trouble.
> Am I not allowed to hold those opinions?
Yes you are allowed to hold these opinions, but you do not have to keep expressing it and repeating yourself just because someone out there is not happy. How about we each express our opinions on the matter only once? Is that so hard?
For the record, I am never happy with "security through obscurity". But it's a matter of the level of risk I can manage/deal with when services are handed over to someone else. Which is why I still use and love Linode, with no plans to leave. I like and appreciate the measures they introduced, even though I would like clearer answers as to what really happened in the "incident." Was it a "hack" against this "customer service portal"? Was it mis-managed credentials? Was this "portal" world accessible, or accessed through someone's compromised machine?
If you are happy with the info Linode has provided, that's fine. Don't belittle someone because they want better answers.
And, yes, when you wote > Hey…. I had a dream last might where aliens from Zabron 9 broke in and stole some accounts.
Should caker sign on and deny this too? you were being just a little silly.
:)
@AgentOfPork:
If you are happy with the info Linode has provided, that's fine. Don't belittle someone because they want better answers.
But the question was about some other provider, and some other platform. That's what I find ridiculous.
@Azathoth:
What else is needed? IP/name/photo of the attacker? Source code of the manager app? Detailed access logs? Names of the victim node owners? Firstborn babies?
I would like to know how the attacker or attackers happened to get access to the customer management portal which allowed him/her/them to reset the root passwords of Linodes.
Either it was a flaw in the management interface, or a valid password was used. If a valid password was used how did the attackers get hold of it?
To anyone who thinks this was a minor attack affecting just a handful of Linodes you should remember we got off lightly because the attacker had very specific targets in mind. It appears it would have been easy enough for the attacker to reset the passwords of every Linode, and from every machine copy off /etc/passwd, /etc/shadow, scan for and copy off any wallet.dat's and .htpasswds, setup a root kit, collect a list of all email addresses the server has ever dealt with, scan for credit card numbers in emails and databases, scan though databases for anything else interesting, redirect any DNS servers to point at a fake drug site or wherever, setup DDOS tools and start attacks against anything.
The attacker could have pre-scripted a whole load of bad stuff and deployed it to every Linode. We could have all been screwed big time.
Was it a hack against a "customer service portal" or was it mis-managed credentials? Both are bad…
Was this portal world accessible (instead of through VPN or other IP restricted access), or accessed through someone's compromised machine? Again, both things are less than optimal…
Just because the incident turned out to be minor, that doesn't equate to the vulnerability being minor.
@AgentOfPork:
Was it a hack against a "customer service portal" or was it mis-managed credentials? Both are bad…
What does it ultimately matter? What really matters is whether or not it can happen again. If it matters that much to you, pick the worst case scenario in your mind and run with it.
@AgentOfPork:
Was this portal world accessible (instead of through VPN or other IP restricted access), or accessed through someone's compromised machine? Again, both things are less than optimal…
Same as above.
@AgentOfPork:
Just because the incident turned out to be minor, that doesn't equate to the vulnerability being minor.
That vulnerability is almost certainly past tense, its your confidence level on how well they learned the lesson and have prepared for the unexpected in the future that matters now. I doubt they will release the gorey details of the prior incident, and in my opinion it would be highly irresponsible for them to do so.
If you've asked Linode directly (i.e. e-mail, support ticket) and they've declined to provide additional details, you are beating the issue to no avail.
@AVonGauss:
What does it ultimately matter? What really matters is whether or not it can happen again. If it matters that much to you, pick the worst case scenario in your mind and run with it.
I disagree. If I pick a worst case scenario of them using "password" for the customer service portal password, then I'd never use Linode again regardless of any claimed "improvements", because that would show that my web hosting company is silly.
> its your confidence level on how well they learned the lesson and have prepared for the unexpected in the future that matters now
I agree, but how shall we quantify this without knowing exactly what the problem was and what they have done to address it?
Just to add, for me it's not a big deal whether we get more information or not, though I'd like more information. I'm not leaving Linode either way.
@nehalem:
I disagree. If I pick a worst case scenario of them using "password" for the customer service portal password, then I'd never use Linode again regardless of any claimed "improvements", because that would show that my web hosting company is silly.
… but, you're already there whether you want to be or not. You take the information you've been given, benefits / cons in general and make the best decision you can.
@nehalem:
I agree, but how shall we quantify this without knowing exactly what the problem was and what they have done to address it?
Same as above, its the past - just because today Linode is uber on the ball (or not) that is no guarantee 6 months or 2 years down the road it won't change - its a constant evaluation process. The incident occurred, they disclosed it to the affected customers (who I don't believe are the ones posting in this thread), they disclosed it publicly, they disclosed the compromised data impacts publicly and they seem to have taken steps to prevent a similar event from occurring in the future. There's nothing specific left for them to do.
I don't mean to sound cold, but it's time to move on in life. If you have a concern great enough to keep this going, you should probably change hosting providers if another gives you a better level of comfort whether it be through historical data to base an opinion on or just the fact they are new to you.
You've already received far more information about this event than you will about the Heartland or Global Payments breaches, and I am pretty sure those are far more impacting in both scope and damage.
The issue is not played out in my mind, I'm not asking for "gorey details", and I'm not asking for proprietary code to be openly posted. I am a professional, working in a business environment. They are professionals, working in a business environment. They have made improvements to the system, that is not in question. But without some level of detail about what happened, there is no way to judge how much the risk was mitigated.
It doesn't have to be a lot. See:
(And yes, I already know how the incidents differ, that's not the point.)
I didn't set about to beat the issue repeatedly, but there were some who were implying that those who originally asked the questions were paranoid, being alarmists, or at least not thinking clearly. Essentially because they refused to agree with another point of view.
You have moved on, great. I have moved on and made decisions based on currently available data as well. That doesn't mean I can't chime in on a discussion, and agree with someone that doesn't agree with you, does it? I don't see value in continuing to ask the same questions in this thread. I never said I did. I only voiced an opinion, which you and others don't agree with.
I'm fine with that. I hope you are as well.
@AgentOfPork:
You have moved on, great. I have moved on and made decisions based on currently available data as well. That doesn't mean I can't chime in on a discussion, and agree with someone that doesn't agree with you, does it? I don't see value in continuing to ask the same questions in this thread. I never said I did. I only voiced an opinion, which you and others don't agree with.
I'm fine with that. I hope you are as well.
I never suggested otherwise, and I can say that without a "[sigh]" tag. This is a forum for Linode customers, primarily read by other Linode customers and especially after 7 pages of the same handful of people reiterating the same points other members such as myself may chime in as well - and may not agree. I would personally have a lot more sympathy even at this point if I thought any of those handful of people were actually a victim of the incident.
@AVonGauss:
I would personally have a lot more sympathy even at this point if I thought any of those handful of people were actually a victim of the incident.
I know one guy that was affected. His loss did affect me actually on a financial and emotional level. If someone shot a gun at a crowd you happened to be in and you didn't get hurt would that be perfectly fine and nothing to worry about?
The incident was actually very minor. It could have taken every single Linode out. I don't want to run wreckless risks with my IT services, shutting my eyes, sticking my fingers in my ears, and going 'Lah Lah Lah, there is no risk' doesn't make the risk go away. People have real companies that depend on this stuff for critical business services like DNS and mail and don't try telling me I should have backup servers because I do and they protect against server failure not deliberate changes to my DNS or mail config by an attacker who gets onto one of my systems.
We need to know. Ignorance isn't the answer.
@sednet:
We need to know. Ignorance isn't the answer.
If you need to know that level of detail, then a VPS host is not the right solution for you.
@glg:
@sednet:We need to know. Ignorance isn't the answer.
If you need to know that level of detail, then a VPS host is not the right solution for you.
That's exactly what I don't understand about this whole thing.
No matter what Linode says, does, promises, has, possesses or implements: at the end of the day, your 'server' is running on a VM on a physical machine you have absolutely no control over.
If that bothers you, the cloud isn't for you.
@ericholtman:
@glg:
@sednet:We need to know. Ignorance isn't the answer.
If you need to know that level of detail, then a VPS host is not the right solution for you.
That's exactly what I don't understand about this whole thing.
No matter what Linode says, does, promises, has, possesses or implements: at the end of the day, your 'server' is running on a VM on a physical machine you have absolutely no control over.
If that bothers you, the cloud isn't for you.
Ok, so maybe I agree with that.
But one way or another. This statement…
> We maintain a strong security policy and aim to communicate openly should it ever be compromised.
… needs to be changed then since they are clearly not communicating openly.
@nehalem:
> We maintain a strong security policy and aim to communicate openly should it ever be compromised.… needs to be changed then since they are clearly not communicating openly.
Yes, they are. They communicated the breach to the world. That's open. To expect nitty gritty details about said breach is ludicrous.
@glg:
@nehalem:
> We maintain a strong security policy and aim to communicate openly should it ever be compromised.… needs to be changed then since they are clearly not communicating openly.
Yes, they are. They communicated the breach to the world. That's open. To expect nitty gritty details about said breach is ludicrous.
"Somebody broke in" isn't, by any stretch of the word, 'open'.
@OverlordQ:
"Somebody broke in" isn't, by any stretch of the word, 'open'.
Which is probably why they said quite a bit more than just, "Somebody broke in". Even if someone published a step-by-step guide to how it was done, included an HD video of the act being done - your Linode would be absolutely no safer than it is today. Instead of focusing on minute details that don't matter, what you should really be concentrating on is whether or not Linode took what learned from the event and did a full review apply ing what they learned to a) all the potential points you're not thinking about, b) all the potential points you have no clue even exist and c) how your own instances are configured and secured.
> what you should really be concentrating on is whether or not Linode took what learned from the event and did a full review apply ing what they learned
What did they learn? What happened in the event? Do a full review of what? Apply what?
How do you propose that we concentrate on something that we have no idea what it is?
@nehalem:
> what you should really be concentrating on is whether or not Linode took what learned from the event and did a full review apply ing what they learnedWhat did they learn? What happened in the event? Do a full review of what? Apply what?
How do you propose that we concentrate on something that we have no idea what it is?
They've already told you what happened, weeks ago. I don't think its a big stretch to say some of the recent enhancements were probably directly inspired by that event. You're berating this to death in my opinion, almost like you're trying to micro manage your service provider which is insane.
> I don't think its a big stretch to say some of the recent enhancements were probably directly inspired by that event
Unlike you, I refuse to speculate.
Also, I don't care about the details. I just want the part about "communicating openly" to be removed.
@glg:
If you need to know that level of detail, then a VPS host is not the right solution for you.
@ericholtman:
That's exactly what I don't understand about this whole thing.
No matter what Linode says, does, promises, has, possesses or implements: at the end of the day, your 'server' is running on a VM on a physical machine you have absolutely no control over.
If that bothers you, the cloud isn't for you.
To use what both of you have said. If Linode was a typical collocation provider and had a break in where someone stole machines would that be good enough information for you? Would you want to know maybe how the thieves broke in, how they were going to prevent it from happening again and generally what is going on?
Lets take it in another direction. Lets say Linode is a managed hosting provider that uses a key to login to servers and that key was compromised and used to login to a server to steal data (this has happened at other providers). Would you want to know how they are going to stop that from happening in the future? How that key was compromised? Why was that key allowed in an area that someone can get to? etc?
By just dismissing this as a "vps/cloud" provider you have to trust you aren't being very realistic. Yes we have to have some trust that Linode will protect its' systems but at the same time we have to have the trust that there will be detailed communication if that trust is breached.
I believe in Linode and its' staff 100% and they are by far the best provider on the market for my needs but this could have been handled way better. It seems Linode took the playbook from Dropbox (blog post and forum post only) instead of being transparent (such as sending an email notifying customers), I'm no longer a dropbox customer because of what they did and there are ample alternatives. Linode likely wanted to reduce panic and general negative press but I think they have lost some trust of their customers. Stuff happens, thinks will break, attacks will occur and it is how they handle it that shows what type of company they are and in my book they have fallen short here.
The topic has been beaten to death and obviously communication isn't happening so you can either live with it which or vote with your wallet. I'm still happy with the service and I trust that Linode is working on the back end so I'll give them the benefit of the doubt. I trust every single team member there and they are some of the hardest working people I know. If there is another issue like this and it is handled the same way then I'll use alternative providers.
@ohkus:
The topic has been beaten to death and obviously communication isn't happening
That's just not true.
@glg:
Yes, they are. They communicated the breach to the world. That's open. To expect nitty gritty details about said breach is ludicrous.
This.
I don't get the problem here. They had a breach. They told us about it. They gave as much detail as was necessary to tell us what happened. Why people are going this lunatic crazy over it for more details is just bizarre to me.
@nehalem:
> I don't think its a big stretch to say some of the recent enhancements were probably directly inspired by that eventUnlike you, I refuse to speculate.
Also, I don't care about the details. I just want the part about "communicating openly" to be removed.
I absolutely agree with you.
I gave up on page 3 of this thread because, quite frankly, these Linode fanboys simple do not listen to reason, nor any opinion other than their own.
(Flame me all you want. It's okay. I'm not here, nor will I be responding any further.)
Now, the Linode team has always been very responsive and I have always praised them in the past. However, with this issue, there has been no information besides the first status post. All requests for any additional detail (anything!) is denied.
What does this mean to mean? This means that the original cause of the issue is somehow embarrassing to Linode. This is why they will not release any information on what exactly happened.
By not releasing such information, I have lost faith in Linode. Had Linode released such information, I would have gotten a laugh, and said "We're all human. Life goes on. Thanks for letting us know."
Linode used to be my home. I had about a dozen clients on Linode. Now, it's down to about 5. I moved to a datacenter with my own dedicated hardware. I have moved a few clients as well. New clients will be placed on my hardware.
It isn't because I have dedicated hardware. It is because I can no longer trust Linode after this breach.
If the Linode team cares at all, they would respond to this thread. They respond to many others. Why is this thread less important?