Accounts hacked via customer support interface: any update?

first official statement's just released http://status.linode.com/2012/03/manage … ident.html">http://status.linode.com/2012/03/manager-security-incident.html

On twitter:

https://twitter.com/#!/rootwyrm/status/ … 1262474242">https://twitter.com/#!/rootwyrm/status/175352831262474242

> Pretty much, if you're a @Linode customer? Your system has almost definitely been hacked and rooted. Because they had a global superuser.

Is this true? Linode says otherwise, however if they admit to something of this extent, they would fear losing their customer base.

as skeptic as I am I'm keen NOT to follow random people's rants on twitter, especially when they are based on 140 chars written by someone else they themselves don't know :-)

My interpretation is that the Linode-side access was limited to what's available via the management UI (and probably a subset at that), but not the hosts themselves. This whole thing could be done with access to the shutdown, change-lish-password, change-root-password, and boot buttons, without having to break into a bunch of different hosts.

By the time anyone notices the reboot, it's done, and the only evidence consists of a Tor exit node IP and a Bitcoin address. Heck of a heist, that's for sure.

That tweet is just moronic.

Linode will probably post a full postmortem report in a few day's time, not in that obscure status site, but in their official blog this time. Security breaches can happen to anyone. What sets responsible companies apart from the rest of the herd is how they handle emergencies like this. I trust that Linode will respond professionally.

About 3 years ago, a budget OpenVZ virtual hosting company with thousands of customers got completely destroyed, all data lost, allegedly because of an unpatched bug in the then-popular HyperVM customer portal. The Indian guy who sold HyperVM committed suicide the next day. What followed was one hell of a mess. But Linode ain't like that, is it?

> Linode will probably post a full postmortem report in a few day's time

Nope.

They just told me they have nothing else to report at this time.

So I will be moving off of Linode and telling everyone I know to do the same. The complete lack of transparency is unacceptable.

@taligent:

Nope.

They just told me they have nothing else to report at this time.
People who have nothing else to report "at this time" often have something new to report after a few days.

"We do not have any plans of releasing any additional information at this time."

That is the exact quote. So I would not be holding your breath.

@taligent:

So I will be moving off of Linode and telling everyone I know to do the same. The complete lack of transparency is unacceptable.
So where will Aunt Betty and your 3 D&D pals be moving to?

Knee jerk reactions with ZERO evidence that there is some big cover up is childish to the extreme.

I'm sure you and your tinfoil hat will have a great time moving to somewhere 100% safe.

I'd believe that quote, after ~7 years of running a node, it gets compromised at the end of february?

Coincidence? Probably not.

An email to customers or at least something on the main page would have been nice. But I certainly wont be leaving it seems Linode responded with those directly affected as fast as they could.

What more can we ask?

At least they have an audit trail!

@vonskippy:

Knee jerk reactions with ZERO evidence that there is some big cover up is childish to the extreme.

I'm sure you and your tinfoil hat will have a great time moving to somewhere 100% safe.

Eh ? At what point did I ever suggest there was a cover up.

I just wish that more information was provided much, much earlier. The same behaviour was exhibited when there was a power outage at Fremont.

This is in no way a reflection of the engineers/admins at Linode who are always quick to respond to questions and supremely helpful.

It's just unacceptable that I should have to read about this on Reddit before hearing from Linode.

are independent security audits worth the money? I mean plenty of IT security companies have been owned.

What bothers me most about this is, assuming the perpetrator was not a Linode employee, Linode's backend customer support interface apparently is accessible over the Internet when it should be locked down and accessible only from designated internal hosts. That's a huge backdoor to every Linode just begging to be opened.

@artagesw:

What bothers me most about this is, assuming the perpetrator was not a Linode employee, Linode's backend customer support interface apparently is accessible over the Internet when it should be locked down and accessible only from designated internal hosts. That's a huge backdoor to every Linode just begging to be opened.

What's the point of saying things like "assuming" and "apparently" and then making a conclusion.

I believe linode aren't transparent enough, they don't keep customers updated very well.

I will definitely think about moving away from Linode, too.

Fact is that I heard about the security breach on Slashdot and then couldn't find anything on Linode's site until browsing the forums – I would expect an e-mail to all of their customers whenever a security breach happens, a proper explanation of why it could happen, and how they are altering their system to prevent any future incidents.

Since they didn't address the last two issues, I would advise any business with valuable data to seriously reconsider Linode and get in touch with them. It's nice that they contacted all those affected, but not enough when security comes into play.

Since the problem was on Linode's side, it's funny that they didn't even tell if they will compensate for the damages.

Looks like Linode have dealt/are dealing with this swiftly enough for it not to be a problem, and you've had a response from one of the most senior people at Linode – yes, it's happened, and that's bad, but they're dealing with it now and that's great.

from the pastebin logs this took about 6 hours to resolve

the question you have to ask yourself is how would every other hosting company/VPS deal with that situation? would it be fixed in that time? would you get a response from the senior management team at that company?

Since Linode's Terms of Service states that

"Subscriber further acknowledges that Linode.com's liability for its own negligence may not in any event exceed an amount equivalent to charges payable by subscriber for services during the period damages occurred."

I guess Linode should be a no-go for any serious business anyway.

@taligent:

Linode will probably post a full postmortem report in a few day's time

Nope.

They just told me they have nothing else to report at this time.

So I will be moving off of Linode and telling everyone I know to do the same. The complete lack of transparency is unacceptable.

Oh look at what the cat dragged in.

There will be a follow up post for sure. They don't have anything else to report for now, currently, at this moment, this very second. Is that clear enough for you?

Given this companies history I'm more than happy to give them the time they need to follow up and make any changes to policy needed. Finally, anyone taking advice from you is clearly in way over their head. So you taking them with you isn't really saying much.

@bcoker:

Oh look at what the cat dragged in.

Seriously ? Is this really necessary or appropriate ?

@bcoker:

There will be a follow up post for sure. They don't have anything else to report for now, currently, at this moment, this very second. Is that clear enough for you?

Care to explain how you know this ? I am just basing my actions from what Linode has told me directly. If you know something I don't then I am sure it would be useful for everyone here.

I don't think I am being unreasonable here. A rogue third party with the ability to instantly get root access to all my Linode servers is a serious issue, no ?

@taligent:

Seriously ? Is this really necessary or appropriate ?

I certainly do. Your knee jerk reaction show's a lack of knowledge of the situation and the industry. Your threat to take customers with you just reinforces that.

Sure, I'll explain how I know. I know because I've been in the business for 17 years. From Floor grunt to Boardroom. If that's not good enough for you then so be it.

They will with 100% certainty create a follow post/article because they know people like me demand to know the details of their after action report. Not to mention the harm it would cause them within the industry as their name was blasted for not doing so. They may lie right thru their teeth about the findings but they will do so either way.

Your concern is not unreasonable. In any way. Yes, it's very unsettling that someone had access to your server that shouldn't have. The reasons why you are stating you are concerned are. Based off ignorance of the situation and knee jerking. It's not an uncommon reaction by some under such circumstances but doesn't make it reasonable or logical.

I'm no linode fanboy here and don't confuse my reasonability and logical approach for weakness of some kind. I'm just not idiot enough to draw my shotgun and start blasting people because I don't know what the whole deal is yet.

Again, if that's not good enough for you then so be it.

Completely agree with bcoker, it is too soon to expect a complete report from Linode on what happened.

I recently moved to Linode and it's a bit disturbing to see it compromised in such a way. They will have to improve their security after this incident, perhaps introduce additional security features on Linode manager.

I does sound like a simple case of slightly bad wording. They've informed people about exactly what has happened, and they did so quickly. More comprehensive information will probably take some time to gather, and doesn't really make sense to release in small bits. That they state they have nothing more to release at the moment shouldn't be read as that they will not do so ever.

Stuff like this happens to every provider now and then unfortunately, and the only thing that really separates the providers in this area is how they deal with it afterwards. At this stage, it seems Linode has done everything right. We know what was done, who was affected, etc, and we knew about it the same day that it happened.

What I want to know now is what steps Linode will take for ensuring this exact scenario will not happen again. It seems official login credentials were used to perform this attack which means that either a support-level employee was careless, or even part of the attack. A possible way to resolve this is adding a higher level person to sign off on stuff like changing root passwords, it would prevent a similar thing from happening again.

But I don't need to know this right this instant. :-)

@taligent:

Seriously ? Is this really necessary or appropriate ?

I think it is entirely appropriate - and in your case, necessary.

James

@compizjoe:

I does sound like a simple case of slightly bad wording. They've informed people about exactly what has happened, and they did so quickly.

No they didn't, they put up a posting on status.linode.com. If there's a serious security breach, they should reach out to me, I shouldn't have to check that site every day to make sure somebody didn't hack in again.

@OverlordQ:

@compizjoe:

I does sound like a simple case of slightly bad wording. They've informed people about exactly what has happened, and they did so quickly.

No they didn't, they put up a posting on status.linode.com. If there's a serious security breach, they should reach out to me, I shouldn't have to check that site every day to make sure somebody didn't hack in again.

According to their investigation, you were not affected by the breach; they did reach out to affected customers. As they say in that post.

Would you also expect to be notified when Billy Joe Bob's linode suffers from a host disk failure, even though you don't know him and don't have any linodes on his host?

Guys, rest assured, Linode executives have called their good friends at Dropbox for some good advice on handling a security lapse and Dropbox are duly advising..

forget it

@Guspaz:

@OverlordQ:

@compizjoe:

I does sound like a simple case of slightly bad wording. They've informed people about exactly what has happened, and they did so quickly.

No they didn't, they put up a posting on status.linode.com. If there's a serious security breach, they should reach out to me, I shouldn't have to check that site every day to make sure somebody didn't hack in again.

According to their investigation, you were not affected by the breach; they did reach out to affected customers. As they say in that post.

Would you also expect to be notified when Billy Joe Bob's linode suffers from a host disk failure, even though you don't know him and don't have any linodes on his host?

First off, all customers were affected, but supposedly only 8 were tampered with further.

Second, no I wouldn't expect to be notifed of that because it doesn't affect me. Critical management infrastructure being broken into, does.

@OverlordQ:

@Guspaz:

@OverlordQ:

No they didn't, they put up a posting on status.linode.com. If there's a serious security breach, they should reach out to me, I shouldn't have to check that site every day to make sure somebody didn't hack in again.

According to their investigation, you were not affected by the breach; they did reach out to affected customers. As they say in that post.

Would you also expect to be notified when Billy Joe Bob's linode suffers from a host disk failure, even though you don't know him and don't have any linodes on his host?

No, because that doesn't affect me. The failure to protect critical management infrastructure does.

Exactly, spot on. I need full details on whether this has anything to do with Linode's control panel. I don't want a Vaserv scenario.

I agree with this. I would like to get this kind of notifications in the same way I get the notifications when I don't pay (i guess they have my email address)

I really love linode but I don't have a better place to go.

But what I really need is the removal of the root password change from the interface and api's.

If I lose it or forget it… Shame on me. Nobody should be able to change it…period.

We need to be sure that nobody can change that not even linode.

If I can do it from an interface then any will (authorize/unauthorized).

@pic.micro23:

But what I really need is the removal of the root password change from the interface and api's.

If I lose it or forget it… Shame on me. Nobody should be able to change it…period.

I agree with this 100%.

This is why I store my coinbits in an old Canopic jar hidden behind the washer in the basement. Lets see Linode (authorized or not) find them there.

@zunzun:

I think it is entirely appropriate - and in your case, necessary.

Really ? Wow. Personal attacks for simply being critical that loyal Linode customers should have been better notified about a major security incident that directly affects them.

This is officially the worst forum I've been to in years.

A few months ago a similar thing happened at Hetzner, and even though I wasn't directly affected, all customers were advised to change their passwords etc. as a precaution.

Just connecting a computer to the internet is a security risk.

If succesful hacking happens to the bods as NASA, then why do some people assume that Linode, or any other hoster is immune?

@artagesw:

@pic.micro23:

But what I really need is the removal of the root password change from the interface and api's.

If I lose it or forget it… Shame on me. Nobody should be able to change it…period.

I agree with this 100%.

You understand that this is just a "nice" feature right? if someone has access to your linode manager account, they can just reboot the node into single user mode, open lish, reset the password and reboot.

Or boot into finnix and do the same thing.

Yes this feature makes it convenient and easier, but its not a security issue.

@taligent:

@zunzun:

I think it is entirely appropriate - and in your case, necessary.

Really ? Wow. Personal attacks for simply being critical that loyal Linode customers should have been better notified about a major security incident that directly affects them.

This is officially the worst forum I've been to in years.

+1

Being rash is one thing, but attacking another person personally shouldn't be accepted (and definitely not endorsed!) by the Linode community. Making wild and potentially hurtful speculations about one's personal life or social status is unacceptable behavior (more so if all he did was demand being informed about security breaches that could easily have compromised his own Linodes.)

@jk4736:

@taligent:

@zunzun:

I think it is entirely appropriate - and in your case, necessary.

Really ? Wow. Personal attacks for simply being critical that loyal Linode customers should have been better notified about a major security incident that directly affects them.

This is officially the worst forum I've been to in years.

+1

Being rash is one thing, but attacking another person personally shouldn't be accepted (and definitely not endorsed!) by the Linode community. Making wild and potentially hurtful speculations about one's personal life or social status is unacceptable behavior (more so if all he did was demand being informed about security breaches that could easily have compromised his own Linodes.)

Think of them as rabid apple fanboys. You attack the product, you get attacked.

In any case, what happened to Linode is SEVERE. I agree an email should have been sent out to ALL clients notifying us of the breach instead of those affected.

Why? It DOES affect ALL of us, not only the 8 that was breached. They had master root access, who knows what they could've done to the rest they didn't have time to dig through?

They didn't have root access.

They had "support representative" access.

If your linode hasn't been rebooted, if your linode's root password hasn't been changed, you haven't been affected.

@rsk:

They didn't have root access.

They had "support representative" access.

If your linode hasn't been rebooted, if your linode's root password hasn't been changed, you haven't been affected.

Hence, if they emailed us explaining in more detail what happened and what did not happen. We would be more informed and less likely to be upset.

I think for such a critical issue, an email notification, even if only pointing to the official "Linode Status Updates" entry, would be in order.

I would much prefer to hear that my provider has had a security breach directly by email from the provider themselves. Or who knows, maybe we'll get that email after the audit has been done.

One also has to consider the unnecessary panic such an announcement might cause among customers.

The Chinese have reportedly hacked the JPL. Everyone is vulnerable. I heard about the breech on Slashdot. That is ok, evidently the staff was already working on the situation.

I'm sure they will do as well as they have in the past.

fos

@rsk:

They didn't have root access.

They had "support representative" access.

If your linode hasn't been rebooted, if your linode's root password hasn't been changed, you haven't been affected.

I love Linode. I really do. I've been a personal customer for many years. I no longer have my own Linode, but still manage about a dozen or so for clients.

I find the above statement confusing. How does 'support representative' access allow access to 8 unrelated accounts (ie - different account holders, different accounts, probably different datacenters), yet not allow access to all other accounts?

To me, and probably to others, if the intruder(s) can attack 8 independent Linodes, then they can attack them all.

Perhaps the Linode team can clarify this?

Also - rumor has it that it was an inside job. Was this the case?

I agree with the sentiment that more information on exactly what happened is needed.

@rainkid:

I find the above statement confusing. How does 'support representative' access allow access to 8 unrelated accounts (ie - different account holders, different accounts, probably different datacenters), yet not allow access to all other accounts?

To me, and probably to others, if the intruder(s) can attack 8 independent Linodes, then they can attack them all.

Perhaps the Linode team can clarify this?

Also - rumor has it that it was an inside job. Was this the case?

I agree with the sentiment that more information on exactly what happened is needed.

As far as I can tell from the information given, normal support level login credentials were used. In other words, no exploit of any kind (Except the human kind) was used, which means they are able to see exactly who was affected through the logs. Yes, the person could in theory have attacked all the nodes, but he or she didn't. There's nothing that needs clarifying about this part specifically, there's no risk of repeat with the same credentials, and nobody else was affected or can be affected in the future as a result of this specific hack.

While a more comprehensive report certainly will be interesting to read, there's no more immediate information that Linode needs to give, the question you asked has already been answered by the official information given. (Or an answer is possible to infer easily).

As far as inside job goes, that would be interesting to know but ultimately doesn't matter that much right now. It's the kind of thing that will be interesting to read in a more comprehensive report of the incident.

@OverlordQ:

Second, no I wouldn't expect to be notifed of that because it doesn't affect me. Critical management infrastructure being broken into, does.

It technically wasn't broken into the same sense as someone hacking it. Someone had login credentials, logged in, did stuff. Linode knows exactly who was affected, and there's no risk at all of this affecting you if you haven't received a notification already. There simply isn't any need for an immediate notification to every customer when we know for sure that only a very specific set was affected and nobody else will be affected in the future.

There is a necessity to give some information to the general public about what had happened, sure, but they did that. Emails should only be used if people are required to take some sort of action. If you want to get status updates in general quickly, subscribe to the feed on status.linode.com. You can probably even get that as an email through some service.

In that case - how did the attacker(s) gain this level of access? How do we know that they no longer have this level of access? If those credentials are no longer valid, how do we know that the attacker cannot acquire new credentials and wreak more havoc?

Essentially, all we are told is 'someone had access, and did bad things. we removed said access.'

Not very informative.

@compizjoe:

@OverlordQ:

Second, no I wouldn't expect to be notifed of that because it doesn't affect me. Critical management infrastructure being broken into, does.

It technically wasn't broken into the same sense as someone hacking it. Someone had login credentials, logged in, did stuff.

Not to add oil to the fire but as far as I can see, there is nothing in the status update to give the impression that it "wasn't broken into the same sense as someone hacking it."

It could easily have been and the status update would still make sense. I think Linode is being deliberately vague at this point so as not to commit either way.

I do know a bit about this world that we are discussing, from both sides of the coin :twisted:

@rainkid:

In that case - how did the attacker(s) gain this level of access? How do we know that they no longer have this level of access? If those credentials are no longer valid, how do we know that the attacker cannot acquire new credentials and wreak more havoc?

Essentially, all we are told is 'someone had access, and did bad things. we removed said access.'

Not very informative.

I suspect finding out exactly how the attackers stumbled upon those credentials will take some more research. But it not reasonable to assume every single credential is also vulnerable. If someone gains unauthorized access to my system using credentials one of my users had written down somewhere, I would, as a system administrator, not then assume the login credentials of every account had become vulnerable. If the attackers did have more extensive access than a simple login credential, then it seems foolish to go through a process where their actions are immediately obvious and logged when they could simply just do whatever they wanted directly.

So either they're so smart they've been able to gain some kind of superprivileged access to the system, yet dumb enough to not use it, or this is simply a case of one login credential getting used by the wrong people. My money would be on the latter.

Why does everyone go out of their way to construct a movie plot threat out of this? :-P

@skn:

Not to add oil to the fire but as far as I can see, there is nothing in the status update to give the impression that it "wasn't broken into the same sense as someone hacking it."

It could easily have been and the status update would still make sense. I think Linode is being deliberately vague at this point so as not to commit either way.

I do know a bit about this world that we are discussing, from both sides of the coin :twisted:

Yes, the status update was a bit vague, I completely agree. After all, it was written only some hours after the event had occured, I suspect it's more a case of limited knowledge at the time rather than a conspiracy to keep people in the dark. :-)

@rainkid:

I find the above statement confusing. How does 'support representative' access allow access to 8 unrelated accounts (ie - different account holders, different accounts, probably different datacenters), yet not allow access to all other accounts?

To me, and probably to others, if the intruder(s) can attack 8 independent Linodes, then they can attack them all.
From my understanding, they got a login/password belonging to one of the Linode support reps - the first level of people who receive your support tickets. Logged in as him, and used the "reset root password for node" option for eight nodes. Everything was logged in audit trail, so Linode knows what happened.

I sure hope I'm not mistaken about it, and we should demand a more detailed report - but in a reasonable time from now. Give'em at least a week to respond before starting a riot.

Somewhat off topic maybe but Slicehost / Rackspace forum just got rooted

~~[http://www.rackspace.com/knowledgecenter/content/slicehost-forum-archive-migration-and-conversion" target="blank">](http://www.rackspace.com/knowledge_cent … conversion">http://www.rackspace.com/knowledge_center/content/slicehost-forum-archive-migration-and-conversion](

Need a new best practices manual.

> We believe an unauthorized party gained access, for a period of time

Why does this sound familiar?

Two hosting companies within a day of each other… :o

The attacker somehow got the login credentials of a Support Rep, yet, knew EXACTLY which eight accounts to target.

How?

It's very likely that the attacker was from the inside, or had MUCH more access than we're told. How else would they know which (independent) accounts to target?

Again, not knocking Linode (they're great and I've told them this many times in the past), but something is amiss here.

@rainkid:

The attacker somehow got the login credentials of a Support Rep, yet, knew EXACTLY which eight accounts to target.

How?

It's very likely that the attacker was from the inside, or had MUCH more access than we're told. How else would they know which (independent) accounts to target?

Again, not knocking Linode (they're great and I've told them this many times in the past), but something is amiss here.

Agreed. Linode may want to give us some more information soon to calm our wild imaginations…

@nehalem:

Agreed. Linode may want to give us some more information soon to calm our wild imaginations…

I'm pretty sure there's nothing more they can really say that would quiet the more active imaginations.

See "birthers", for reference.

@nehalem:

Agreed. Linode may want to give us some more information soon to calm our wild imaginations…
My neighbors friend has an uncle that knew someone that read on Hacker News that Linode will be sending out partial rolls of generic tinfoil in the next billing cycle, and will include not three, but four unique ways to fold them into tinfoil hats guaranteed to protect you from the outer space Nargles and the local FBI.

@rainkid:

The attacker somehow got the login credentials of a Support Rep, yet, knew EXACTLY which eight accounts to target.
All of the victims were Bitcoin dealers. Machines running Bitcoin software are readily identifiable through a port scan or through transaction records with other Bitcoin machines. Once you know which IP addresses to target, it would be trivially easy for someone who has the credentials of a support rep to figure out exactly which accounts to break into.

@hybinet:

All of the victims were Bitcoin dealers. Machines running Bitcoin software are readily identifiable through a port scan or through transaction records with other Bitcoin machines.

See what you did? Injected actual facts into the tinfoil hat party, and totally killed it!

Any updates with more details on what happened?

How is it that more people are not demanding to know more details on this? Did I just miss the newer post(s)?

It seems to me that all the responses from people saying "they need more time" have pretty much lost their weight.

I absolutely love Linode but the way they just ignore the forum posts about this, as if hoping it just goes away is VERY bothersome.

I hope others join me in continuing to pursue this subject.

I want to keep loving Linode and being a faithful customer for years to come but for that to happen I have to be able to trust them and for that to happen, they have to be open, good or bad, just like we all learned while growing up.

Please give us the details!

Linode is extremely secretive. It always has been. I lost a bet that they acknowledged the break in, in the first place, i didn't think they would. I'm very confident you won't hear anything else from them about the subject.

It's their MO.

@chesty:

Linode is extremely secretive. It always has been. I lost a bet that they acknowledged the break in, in the first place, i didn't think they would. I'm very confident you won't hear anything else from them about the subject.

It's their MO.

That is disapointing to hear, for two reasons.

The first reason is obvious. I want information and they aren't sharing.

The second reason is that, if what you say is true, it also makes them liars.

In the statement they did release, they open by saying…

> Ensuring the security of our platform is our top priority. We maintain a strong security policy and aim to communicate openly should it ever be compromised. Thus, we are posting to describe a recent incident affecting the Linode Manager.

If you have a policy of openness, remain open about these things, otherwise just change the policy.

If you want to remain quiet in these situations, at least come out and say so. It would be much better to just tell us "Something happened, those affected have been notified, we will discuss nothing further on the matter.". But that isn't what they did, they made a public announcement basically claiming that they have an ongoing policy of openness.

I hope at some point someone high up just finally comes out and openly discusses this OR just tell us that they have no intentions of doing so. Ignoring something to make it go away is not the proper course of action.

A large part of it might be that there's nothing more to say: a customer service password was brute-forced, someone used it to "recover" access to a handful of accounts, and that's what happened. There were contributing factors that made this easier than it should have been (see 1 and 2, as well as the accountholders' failure to adequately compartmentalize sensitive data in their own systems).

Is there anything in specific beyond this that ought to be disclosed, and doesn't fall into the realm of Things That Would Jeopardize FBI Investigations or Things That Would Violate Privacy Policies And Laws?

I, too, would like to know more, but I don't know what practical use that knowledge would be. If Linode had more of my personal information, I'd probably demand credit reporting service or something, but they don't, and I already have it.

would any of this explain why I can't access either webmin or ftp?

@worldviewpr:

would any of this explain why I can't access either webmin or ftp?
nope..

@hoopycat:

A large part of it might be that there's nothing more to say: a customer service password was brute-forced, someone used it to "recover" access to a handful of accounts

The story on the register claims that Linode network equipment got p0wned. My guess is that network sniffers were then used to grab a customer support password or maybe an authentication cookie.

I don't like guessing. People that trust me have minor stuff on Linode because I recommended it. I also have some personal stuff on Linode. I can't estimate risk on guesswork alone.

So what do I do if I can't trust cloud stuff, virtual machines, or even dedicated machines that are plugged into someone else's remote management setup? Go to all the expense and hassle of setting up a server myself and getting it colocated?

@Typo:

I hope at some point someone high up just finally comes out and openly discusses this OR just tell us that they have no intentions of doing so.

I opened a support ticket to asked about this. I was very politely told by Caker himself that they would not be giving out any more information.

If anyone else wants to open a support ticket to ask the same question feel free. Maybe they will cave if they get 100 of these things.

@sednet:

I opened a support ticket to asked about this. I was very politely told by Caker himself that they would not be giving out any more information.

If anyone else wants to open a support ticket to ask the same question feel free. Maybe they will cave if they get 100 of these things.
That just sucks. How is it that more people aren't demanding information, especially after it was promised by caker himself on page 10 of this post in reply to a request for more transparency.

I just cannot believe the policy is to just ignore it after that kind of public promise to give information.

Here is the conversation from the other post….
@caker:

@scaredpoet:

Between the furor over the Bitcoin incident and the beating linode took over its lack of cooperation in the lowendbox/lowendtalk DDoS incident, I for one am re-evaluating whether I want my VPSes here or whether it's time to move on. Taken in total with my own experiences with Linode support in the past, and the attitude towards things like IPv6 migration, I think it's fair to say that Linode is quickly earning a reputation for being not as customer friendly as they were once thought to be.
Very much appreciate your comments.

Since last week, we've been completely consumed with evaluating, discussing, debating, planning, etc, ways in which we can do better. This was a learning experience for us and Linode will only improve because of it. Hoping to have an announcement soon covering the results of these efforts.

With regards to the lowendbox thing - we handled it the same way we handle all network attacks. The forum post from those guys had ZERO effect on how it was handled. A threat of a DDoS never provokes preemptive action from us, unless the customer requests it. We left the forum post there in order to BE more transparent, if that makes sense…

Not sure what you mean regarding IPv6. What attitude? We've worked hard to make native IPv6 available to you guys, which it is now in all six of our facilities.

Thanks,

-Chris

@Typo:

That just sucks. How is it that more people aren't demanding information, especially after it was promised by caker himself on page 10 of this post in reply to a request for more transparency.

I just cannot believe the policy is to just ignore it after that kind of public promise to give information.

Here is the conversation from the other post….
@caker:

@scaredpoet:

Between the furor over the Bitcoin incident and the beating linode took over its lack of cooperation in the lowendbox/lowendtalk DDoS incident, I for one am re-evaluating whether I want my VPSes here or whether it's time to move on. Taken in total with my own experiences with Linode support in the past, and the attitude towards things like IPv6 migration, I think it's fair to say that Linode is quickly earning a reputation for being not as customer friendly as they were once thought to be.
Very much appreciate your comments.

Since last week, we've been completely consumed with evaluating, discussing, debating, planning, etc, ways in which we can do better. This was a learning experience for us and Linode will only improve because of it. Hoping to have an announcement soon covering the results of these efforts.

The following blog posts seem to cover "the results of these efforts":

http://blog.linode.com/2012/04/05/event … rotection/">http://blog.linode.com/2012/04/05/events-rss-feed-emails-and-profile-protection/

http://blog.linode.com/2012/04/05/linod … rotection/">http://blog.linode.com/2012/04/05/linode-manager-brute-force-protection/

@retrograde inversion:

The following blog posts seem to cover "the results of these efforts":

http://blog.linode.com/2012/04/05/event … rotection/">http://blog.linode.com/2012/04/05/events-rss-feed-emails-and-profile-protection/

http://blog.linode.com/2012/04/05/linod … rotection/">http://blog.linode.com/2012/04/05/linode-manager-brute-force-protection/

For one, no post I have seen has been made saying for sure what methods were used to gain access.

I am not going to just assume that a couple blog entries made like a month later about new features related to some of the most basic security features available to protect sensitive areas of a website are a response to what happened nor will I just assume that those two posts cleared everything up because I still don't even know the extent of what happened or if this is enough to make sure it cannot happen again.

I'm not sure how you came to the conclusion that those posts were anything even close to the promised announcement covering the results of the efforts they took to make sure this cannot happen. It wasn't even mentioned unless I missed it.

@Typo:

For one, no post I have seen has been made saying for sure what methods were used to gain access.

I'm not replying in defense of Linode, this post would be poor representation, but I can't help but to reply. What are you expecting them to post? A blow-by-blow or step guide to how the individual or group was able to accomplish the break in? Really? That's the last thing you should want them to post. Sure, I have an idle curiosity as well, but what it all boils down to is whether or not I have confidence in the fact it won't happen in the future. They've made a series of changes recently, presumably to prevent this from occurring again, its up to you to decide.

I know this falls in to the category of blaming the victim, but as far as the Bitcoin concept and operators go… 1) Maybe this is another example of why its not such a good idea. 2) If you're going to store sensitive data that is accesible via the Internet, you darn well better make sure its properly secured - and yes, that includes if someone has access to your Linode Manager account.

I simply want the announcement that caker said would be forthcoming.

I don't expect every single little detail but the basics of what happened and how they have ensured it won't happen again would be fantastic.

It is not an unusual request, especially considering the nature of the situation. This company uses a completely proprietary manager which can, if exploited, get around the other security measures which may be in place to protect our nodes. Some of the users here have very sensitive data and/or clients with sensitive data and everyone in that picture deserves peace of mind given back after this type of failure in security.

I want to make it clear that I love linode. I loved this place even before I was a client, I think the service and setup just rocks. I would just like to see a little more openness when a mistake is made.

I think even a post saying something along the lines of "We have finished investigating the matter and have ensured that this cannot happen any more, sorry again, here's a free beer".

@Typo:

It is not an unusual request, especially considering the nature of the situation.

Its not unusual, but you also have to have a realistic expectation and beating it to death does not help. They have been responding, you just don't seem satisfied.

@Typo:

This company uses a completely proprietary manager which can, if exploited, get around the other security measures which may be in place to protect our nodes.

What does proprietary have to do with anything here? However they gained access, once you have access through Linode's proprietary system or your favorite open source Xen manager - you have access.

@Typo:

Some of the users here have very sensitive data and/or clients with sensitive data and everyone in that picture deserves peace of mind given back after this type of failure in security.

Then those users (and I say that like that because you didn't include yourself in your own statement) should be implementing multiple measures to ensure that data remains secure. If you give me access to your Linode Manager account and you're worried about me accessing sensitive data rather than just deleting data, you are doing something wrong.

Security is a matter of multiple layers, and none of them are absolutely effective. You're only real hope is to make it more painful than the gain and/or slow them down enough until more direct measures can be implemented. I imagine this is why they implemented the e-mail alerts. Though, I'm not sure why it doesn't send an e-mail alert when you change the alert setting from enabled to disabled.

@AVonGauss:

What does proprietary have to do with anything here? However they gained access, once you have access through Linode's proprietary system or your favorite open source Xen manager - you have access.
It has a lot to do with it, with "my favorite open source" app, everyone can see the code, if its unsafe, its usually discovered and reported and fixed. In this case, we just have to trust them.

@AVonGauss:

Its not unusual, but you also have to have a realistic expectation and beating it to death does not help. They have been responding, you just don't seem satisfied.
I don't feel I've beaten anything to death. I voiced a concern in reply to a post (in 2 different threads granted) and have responded to replies to those posts. Also, nobody with any authority has responded to any of the posts regarding this, nor has any announcement relating to it been released since we were told to expect one. I fail to see where "they have been responding" unless your referring to the recent security enhancements which again would just be assumption.

I just personally think that this long after the incident, the situation should be resolved and the announcement should have been made.

I want to say again. I love this place and think its a great service and I am in no way trying to start an argument or troll.

@Typo:

@AVonGauss:

What does proprietary have to do with anything here? However they gained access, once you have access through Linode's proprietary system or your favorite open source Xen manager - you have access.
It has a lot to do with it, with "my favorite open source" app, everyone can see the code, if its unsafe, its usually discovered and reported and fixed. In this case, we just have to trust them.

@AVonGauss:

Its not unusual, but you also have to have a realistic expectation and beating it to death does not help. They have been responding, you just don't seem satisfied.
I don't feel I've beaten anything to death. I voiced a concern in reply to a post (in 2 different threads granted) and have responded to replies to those posts. Also, nobody with any authority has responded to any of the posts regarding this, nor has any announcement relating to it been released since we were told to expect one. I fail to see where "they have been responding" unless your referring to the recent security enhancements which again would just be assumption.

I just personally think that this long after the incident, the situation should be resolved and the announcement should have been made.

I want to say again. I love this place and think its a great service and I am in no way trying to start an argument or troll.

If Linode has chosen to investigate with the authorities then they may be legally obliged not to respond.

@tonymallin:

If Linode has chosen to investigate with the authorities then they may be legally obliged not to respond.
Except they could say "Due to ongoing investigations, we're unable to comment at this time."

Instead, they just ignore all the chatter.

My guess, people will start voting on their "opaque transparency" with their feet (or should that be wallet?).

They still list their "Marketing Guru" as open on their "We're Hiring" page, so no big surprise that there isn't any real change in how they handle stuff like this.

@vonskippy:

Instead, they just ignore all the chatter.

It doesn't matter where you draw the line here, someone is going to be unhappy.

Me? I'm happy.

@ericholtman:

@vonskippy:

Instead, they just ignore all the chatter.

It doesn't matter where you draw the line here, someone is going to be unhappy.

Me? I'm happy.

I'm not really happy. It appears linode lost a customer support interface account that lets whoever uses it change the root password on any linode. We got lucky that the attacker appeared to be one guy with an interest in BitCoins. It could easily have been a hundred plus script kiddie hackers who could have prepared tools to loot these machines in advance. With one big server outside they could have copied off everything that looked remotely interesting from every Linode and sorted though it later.

Do these posts actually get read by the Linode hierarchy though? Otherwise this is just flogging a dead horse…

@Mr Nod:

Do these posts actually get read by the Linode hierarchy though? Otherwise this is just flogging a dead horse…

Every post on these forums is read (or at least skimmed) by Linode (don't know if Caker has the time to read them all himself).

@Mr Nod:

Do these posts actually get read by the Linode hierarchy though? Otherwise this is just flogging a dead horse…

Unless Caker has had a personality transplant in the last few years he reads and cares about what people say here.

I wish I knew what he really thought about this. He must know that Linux hobbyists are an awkward bunch and won't just forget about something as serious as this.

> We maintain a strong security policy and aim to communicate openly should it ever be compromised.

For me and some of the others (I'm assuming), the little official information we got could in no way be construed as Linode "communicating openly" about what happened.

@sednet:

Unless Caker has had a personality transplant in the last few years he reads and cares about what people say here.
Rackspace bought his personality to use as the basis for their Fanatical Support. Now he's a grumpy BOFH. :(

A certain competitor has been emailing it's customers explaining a security issue that existed and has now been fixed. Perhaps Linode are experiencing the same issue, but haven't found the fix yet.

@graq:

A certain competitor has been emailing it's customers explaining a security issue that existed and has now been fixed. Perhaps Linode are experiencing the same issue, but haven't found the fix yet.
I kind of assumed it was something in the manager but like every other idea we all throw out its just guess work and assumption because nobody who knows is talking.

How about more details just in case your hypothesis proves true?

@Typo:

How about more details just in case your hypothesis proves true?

Hey…. I had a dream last might where aliens from Zabron 9 broke in and stole some accounts.

Should caker sign on and deny this too?

@ericholtman:

@Typo:

How about more details just in case your hypothesis proves true?

Hey…. I had a dream last might where aliens from Zabron 9 broke in and stole some accounts.

Should caker sign on and deny this too?

You don't need to be silly. You indicated that you are happy with the little information you got. We get it. Now, there are others who are not happy with the little information they got, and I think reasonably so.

Basically you're not helping.

@nehalem:

You don't need to be silly.

Some other competitor reports some other security breach on some other platform, and someone here wants a response to that, and I'm being silly?

@ericholtman:

@nehalem:

You don't need to be silly.

Some other competitor reports some other security breach on some other platform, and someone here wants a response to that, and I'm being silly?
I asked for more information since the info he gave was seriously lacking. If he was honestly curious if the security issues were at all related, which I already sort of pointed out that I don't think it is, then he will need to provide more information.

We have enough lack of information in this thread to go around, I was just trying to lessen it a bit.

You are obviously posting simply to start trouble while we have an honest desire to get answers that were promised us and are directly related to the safety of our vps's.

Yes, YOU are being silly.

I don't understand this thread.

http://status.linode.com/2012/03/manage … ident.html">http://status.linode.com/2012/03/manager-security-incident.html

What else is needed? IP/name/photo of the attacker? Source code of the manager app? Detailed access logs? Names of the victim node owners? Firstborn babies?

@Typo:

You are obviously posting simply to start trouble while we have an honest desire to get answers that were promised us and are directly related to the safety of our vps's.

I am not posting "just to start trouble".

I am posting to say that I do think we've gotten answers. I think we've gotten all the answers we're going to get. And I am satisfied with those answers.

Am I not allowed to hold those opinions?

> What else is needed? IP/name/photo of the attacker? Source code of the manager app? Detailed access logs? Names of the victim node owners? Firstborn babies?

For me, I want to know how the attacker accessed the web-based Linode customer service portal. Was it brute-force? Was it phishing? Was it a vulnerability in the portal itself? Was it an inside job? How did the attacker target the bitcoin people so quickly? I would like concrete answers to those questions without the speculation.

> I am not posting "just to start trouble".

To me, it seems you are. You already posted:

> It doesn't matter where you draw the line here, someone is going to be unhappy.

Me? I'm happy.

And we get it. You are happy with the response you have gotten. Good for you. And you expressed that you are satisfied. If everytime someone who is not happy posts somethings, and you reply by saying you are happy, then yes, it does seem you are trying to start trouble.

> Am I not allowed to hold those opinions?

Yes you are allowed to hold these opinions, but you do not have to keep expressing it and repeating yourself just because someone out there is not happy. How about we each express our opinions on the matter only once? Is that so hard?

@ericholtman: You have voiced your opinion, no problem. But you also made fun of someone else who has voiced theirs, only because they disagree with you.

For the record, I am never happy with "security through obscurity". But it's a matter of the level of risk I can manage/deal with when services are handed over to someone else. Which is why I still use and love Linode, with no plans to leave. I like and appreciate the measures they introduced, even though I would like clearer answers as to what really happened in the "incident." Was it a "hack" against this "customer service portal"? Was it mis-managed credentials? Was this "portal" world accessible, or accessed through someone's compromised machine?

If you are happy with the info Linode has provided, that's fine. Don't belittle someone because they want better answers.

And, yes, when you wote > Hey…. I had a dream last might where aliens from Zabron 9 broke in and stole some accounts.

Should caker sign on and deny this too? you were being just a little silly. :)

@AgentOfPork:

If you are happy with the info Linode has provided, that's fine. Don't belittle someone because they want better answers.

But the question was about some other provider, and some other platform. That's what I find ridiculous.

@Azathoth:

What else is needed? IP/name/photo of the attacker? Source code of the manager app? Detailed access logs? Names of the victim node owners? Firstborn babies?

I would like to know how the attacker or attackers happened to get access to the customer management portal which allowed him/her/them to reset the root passwords of Linodes.

Either it was a flaw in the management interface, or a valid password was used. If a valid password was used how did the attackers get hold of it?

To anyone who thinks this was a minor attack affecting just a handful of Linodes you should remember we got off lightly because the attacker had very specific targets in mind. It appears it would have been easy enough for the attacker to reset the passwords of every Linode, and from every machine copy off /etc/passwd, /etc/shadow, scan for and copy off any wallet.dat's and .htpasswds, setup a root kit, collect a list of all email addresses the server has ever dealt with, scan for credit card numbers in emails and databases, scan though databases for anything else interesting, redirect any DNS servers to point at a fake drug site or wherever, setup DDOS tools and start attacks against anything.

The attacker could have pre-scripted a whole load of bad stuff and deployed it to every Linode. We could have all been screwed big time.

Exactly my point(s)! Nobody is bashing the company, just concerned customers questioning the company. We would really like to know:

Was it a hack against a "customer service portal" or was it mis-managed credentials? Both are bad…

Was this portal world accessible (instead of through VPN or other IP restricted access), or accessed through someone's compromised machine? Again, both things are less than optimal…

Just because the incident turned out to be minor, that doesn't equate to the vulnerability being minor.

@AgentOfPork:

Was it a hack against a "customer service portal" or was it mis-managed credentials? Both are bad…

What does it ultimately matter? What really matters is whether or not it can happen again. If it matters that much to you, pick the worst case scenario in your mind and run with it.

@AgentOfPork:

Was this portal world accessible (instead of through VPN or other IP restricted access), or accessed through someone's compromised machine? Again, both things are less than optimal…

Same as above.

@AgentOfPork:

Just because the incident turned out to be minor, that doesn't equate to the vulnerability being minor.

That vulnerability is almost certainly past tense, its your confidence level on how well they learned the lesson and have prepared for the unexpected in the future that matters now. I doubt they will release the gorey details of the prior incident, and in my opinion it would be highly irresponsible for them to do so.

If you've asked Linode directly (i.e. e-mail, support ticket) and they've declined to provide additional details, you are beating the issue to no avail.

@AVonGauss:

What does it ultimately matter? What really matters is whether or not it can happen again. If it matters that much to you, pick the worst case scenario in your mind and run with it.

I disagree. If I pick a worst case scenario of them using "password" for the customer service portal password, then I'd never use Linode again regardless of any claimed "improvements", because that would show that my web hosting company is silly.

> its your confidence level on how well they learned the lesson and have prepared for the unexpected in the future that matters now

I agree, but how shall we quantify this without knowing exactly what the problem was and what they have done to address it?

Just to add, for me it's not a big deal whether we get more information or not, though I'd like more information. I'm not leaving Linode either way.

@nehalem:

I disagree. If I pick a worst case scenario of them using "password" for the customer service portal password, then I'd never use Linode again regardless of any claimed "improvements", because that would show that my web hosting company is silly.

… but, you're already there whether you want to be or not. You take the information you've been given, benefits / cons in general and make the best decision you can.

@nehalem:

I agree, but how shall we quantify this without knowing exactly what the problem was and what they have done to address it?

Same as above, its the past - just because today Linode is uber on the ball (or not) that is no guarantee 6 months or 2 years down the road it won't change - its a constant evaluation process. The incident occurred, they disclosed it to the affected customers (who I don't believe are the ones posting in this thread), they disclosed it publicly, they disclosed the compromised data impacts publicly and they seem to have taken steps to prevent a similar event from occurring in the future. There's nothing specific left for them to do.

I don't mean to sound cold, but it's time to move on in life. If you have a concern great enough to keep this going, you should probably change hosting providers if another gives you a better level of comfort whether it be through historical data to base an opinion on or just the fact they are new to you.

You've already received far more information about this event than you will about the Heartland or Global Payments breaches, and I am pretty sure those are far more impacting in both scope and damage.

[sigh]

The issue is not played out in my mind, I'm not asking for "gorey details", and I'm not asking for proprietary code to be openly posted. I am a professional, working in a business environment. They are professionals, working in a business environment. They have made improvements to the system, that is not in question. But without some level of detail about what happened, there is no way to judge how much the risk was mitigated.

It doesn't have to be a lot. See:

~~[http://status.linode.com/2011/08/fremont-power-outage-rfo.html" target="_blank">](http://status.linode.com/2011/08/fremon … e-rfo.html">http://status.linode.com/2011/08/fremont-power-outage-rfo.html](

(And yes, I already know how the incidents differ, that's not the point.)

I didn't set about to beat the issue repeatedly, but there were some who were implying that those who originally asked the questions were paranoid, being alarmists, or at least not thinking clearly. Essentially because they refused to agree with another point of view.

You have moved on, great. I have moved on and made decisions based on currently available data as well. That doesn't mean I can't chime in on a discussion, and agree with someone that doesn't agree with you, does it? I don't see value in continuing to ask the same questions in this thread. I never said I did. I only voiced an opinion, which you and others don't agree with.

I'm fine with that. I hope you are as well.

@AgentOfPork:

You have moved on, great. I have moved on and made decisions based on currently available data as well. That doesn't mean I can't chime in on a discussion, and agree with someone that doesn't agree with you, does it? I don't see value in continuing to ask the same questions in this thread. I never said I did. I only voiced an opinion, which you and others don't agree with.

I'm fine with that. I hope you are as well.

I never suggested otherwise, and I can say that without a "[sigh]" tag. This is a forum for Linode customers, primarily read by other Linode customers and especially after 7 pages of the same handful of people reiterating the same points other members such as myself may chime in as well - and may not agree. I would personally have a lot more sympathy even at this point if I thought any of those handful of people were actually a victim of the incident.

@AVonGauss:

I would personally have a lot more sympathy even at this point if I thought any of those handful of people were actually a victim of the incident.

I know one guy that was affected. His loss did affect me actually on a financial and emotional level. If someone shot a gun at a crowd you happened to be in and you didn't get hurt would that be perfectly fine and nothing to worry about?

The incident was actually very minor. It could have taken every single Linode out. I don't want to run wreckless risks with my IT services, shutting my eyes, sticking my fingers in my ears, and going 'Lah Lah Lah, there is no risk' doesn't make the risk go away. People have real companies that depend on this stuff for critical business services like DNS and mail and don't try telling me I should have backup servers because I do and they protect against server failure not deliberate changes to my DNS or mail config by an attacker who gets onto one of my systems.

We need to know. Ignorance isn't the answer.

@sednet:

We need to know. Ignorance isn't the answer.

If you need to know that level of detail, then a VPS host is not the right solution for you.

@glg:

@sednet:

We need to know. Ignorance isn't the answer.

If you need to know that level of detail, then a VPS host is not the right solution for you.

That's exactly what I don't understand about this whole thing.

No matter what Linode says, does, promises, has, possesses or implements: at the end of the day, your 'server' is running on a VM on a physical machine you have absolutely no control over.

If that bothers you, the cloud isn't for you.

@ericholtman:

@glg:

@sednet:

We need to know. Ignorance isn't the answer.

If you need to know that level of detail, then a VPS host is not the right solution for you.

That's exactly what I don't understand about this whole thing.

No matter what Linode says, does, promises, has, possesses or implements: at the end of the day, your 'server' is running on a VM on a physical machine you have absolutely no control over.

If that bothers you, the cloud isn't for you.

Ok, so maybe I agree with that.

But one way or another. This statement…

> We maintain a strong security policy and aim to communicate openly should it ever be compromised.

… needs to be changed then since they are clearly not communicating openly.

@nehalem:

> We maintain a strong security policy and aim to communicate openly should it ever be compromised.

… needs to be changed then since they are clearly not communicating openly.

Yes, they are. They communicated the breach to the world. That's open. To expect nitty gritty details about said breach is ludicrous.

@glg:

@nehalem:

> We maintain a strong security policy and aim to communicate openly should it ever be compromised.

… needs to be changed then since they are clearly not communicating openly.

Yes, they are. They communicated the breach to the world. That's open. To expect nitty gritty details about said breach is ludicrous.

"Somebody broke in" isn't, by any stretch of the word, 'open'.

@OverlordQ:

"Somebody broke in" isn't, by any stretch of the word, 'open'.

Which is probably why they said quite a bit more than just, "Somebody broke in". Even if someone published a step-by-step guide to how it was done, included an HD video of the act being done - your Linode would be absolutely no safer than it is today. Instead of focusing on minute details that don't matter, what you should really be concentrating on is whether or not Linode took what learned from the event and did a full review apply ing what they learned to a) all the potential points you're not thinking about, b) all the potential points you have no clue even exist and c) how your own instances are configured and secured.

> what you should really be concentrating on is whether or not Linode took what learned from the event and did a full review apply ing what they learned

What did they learn? What happened in the event? Do a full review of what? Apply what?

How do you propose that we concentrate on something that we have no idea what it is?

@nehalem:

> what you should really be concentrating on is whether or not Linode took what learned from the event and did a full review apply ing what they learned

What did they learn? What happened in the event? Do a full review of what? Apply what?

How do you propose that we concentrate on something that we have no idea what it is?

They've already told you what happened, weeks ago. I don't think its a big stretch to say some of the recent enhancements were probably directly inspired by that event. You're berating this to death in my opinion, almost like you're trying to micro manage your service provider which is insane.

> I don't think its a big stretch to say some of the recent enhancements were probably directly inspired by that event

Unlike you, I refuse to speculate.

Also, I don't care about the details. I just want the part about "communicating openly" to be removed.

@glg:

If you need to know that level of detail, then a VPS host is not the right solution for you.

@ericholtman:

That's exactly what I don't understand about this whole thing.

No matter what Linode says, does, promises, has, possesses or implements: at the end of the day, your 'server' is running on a VM on a physical machine you have absolutely no control over.

If that bothers you, the cloud isn't for you.

To use what both of you have said. If Linode was a typical collocation provider and had a break in where someone stole machines would that be good enough information for you? Would you want to know maybe how the thieves broke in, how they were going to prevent it from happening again and generally what is going on?

Lets take it in another direction. Lets say Linode is a managed hosting provider that uses a key to login to servers and that key was compromised and used to login to a server to steal data (this has happened at other providers). Would you want to know how they are going to stop that from happening in the future? How that key was compromised? Why was that key allowed in an area that someone can get to? etc?

By just dismissing this as a "vps/cloud" provider you have to trust you aren't being very realistic. Yes we have to have some trust that Linode will protect its' systems but at the same time we have to have the trust that there will be detailed communication if that trust is breached.

I believe in Linode and its' staff 100% and they are by far the best provider on the market for my needs but this could have been handled way better. It seems Linode took the playbook from Dropbox (blog post and forum post only) instead of being transparent (such as sending an email notifying customers), I'm no longer a dropbox customer because of what they did and there are ample alternatives. Linode likely wanted to reduce panic and general negative press but I think they have lost some trust of their customers. Stuff happens, thinks will break, attacks will occur and it is how they handle it that shows what type of company they are and in my book they have fallen short here.

The topic has been beaten to death and obviously communication isn't happening so you can either live with it which or vote with your wallet. I'm still happy with the service and I trust that Linode is working on the back end so I'll give them the benefit of the doubt. I trust every single team member there and they are some of the hardest working people I know. If there is another issue like this and it is handled the same way then I'll use alternative providers.

@ohkus:

The topic has been beaten to death and obviously communication isn't happening

That's just not true.

@glg:

Yes, they are. They communicated the breach to the world. That's open. To expect nitty gritty details about said breach is ludicrous.
This.

I don't get the problem here. They had a breach. They told us about it. They gave as much detail as was necessary to tell us what happened. Why people are going this lunatic crazy over it for more details is just bizarre to me.

@nehalem:

> I don't think its a big stretch to say some of the recent enhancements were probably directly inspired by that event

Unlike you, I refuse to speculate.

Also, I don't care about the details. I just want the part about "communicating openly" to be removed.

I absolutely agree with you.

I gave up on page 3 of this thread because, quite frankly, these Linode fanboys simple do not listen to reason, nor any opinion other than their own.

(Flame me all you want. It's okay. I'm not here, nor will I be responding any further.)

Now, the Linode team has always been very responsive and I have always praised them in the past. However, with this issue, there has been no information besides the first status post. All requests for any additional detail (anything!) is denied.

What does this mean to mean? This means that the original cause of the issue is somehow embarrassing to Linode. This is why they will not release any information on what exactly happened.

By not releasing such information, I have lost faith in Linode. Had Linode released such information, I would have gotten a laugh, and said "We're all human. Life goes on. Thanks for letting us know."

Linode used to be my home. I had about a dozen clients on Linode. Now, it's down to about 5. I moved to a datacenter with my own dedicated hardware. I have moved a few clients as well. New clients will be placed on my hardware.

It isn't because I have dedicated hardware. It is because I can no longer trust Linode after this breach.

If the Linode team cares at all, they would respond to this thread. They respond to many others. Why is this thread less important?

@rainkid:

I gave up on page 3 of this thread because, quite frankly, these Linode fanboys simple do not listen to reason, nor any opinion other than their own.
Pot, meet Kettle. Kettle, meet Pot.

@rainkid:

I gave up on page 3 of this thread because, quite frankly, these Linode fanboys simple do not listen to reason, nor any opinion other than their own.

I've been partisipating in these forums on and off for about 8 years now, I've had various numbers of Linodes on and off during that time starting with a 80Meg UML machine. I've had very few problems and the ones I have had were resolved quickly. However I have to say the above comment is spot on. There are 'contributers' on this board that will quickly shout down any perceived criticism of linode or it's service regardless of the validity of the criticism.

IPv6 support was a good example, before IPv6 was deployed at linode and after it was deployed at many other providers anyone that asked for it on these forums was told they didn't need it and they were insulted for asking.

There were a few examples of people who canceled their accounts and didn't get a refund for the unused part. These people were soundly mocked for no good reason.

There was one chinese guy who got all frustrated at Linode wanting copies of the front and back ( PCI rules anyone? ) of his credit card and his passport. He was told he must be a scammer or somesuch thing on this forum based on nothing but his country of origin. Sad, that was a potential customer.

In this latest incident Linode screwed up, there is no denying it. No doubt Linode are doing everything they can to fix the situation but it doesn't not distract from the point that they did screw up. Yet the fanboys deny it, they make excuses, they try to derail any criticism.

These people are just a minority of forum contributers but they are vocal. I don't believe these people are sockpuppets. I believe these people just have a flawed view of reality caused by some kind of cognitive bias. I would not trust these people with a root account on any machine running anything I care about.

TL;DR - Linode screwed up, fanboys are unable to accept this so keep trying to derail this thread.

@sednet:

In this latest incident Linode screwed up, there is no denying it. No doubt Linode are doing everything they can to fix the situation but it doesn't not distract from the point that they did screw up. Yet the fanboys deny it, they make excuses, they try to derail any criticism.

Did they screw up by allowing some Linodes to get compromised?

Probably (although if it was inside job, there's nothing that could be done about it).

Did they screw up by not providing second-by-second analysis, real-time video of the breakin, names addresses and birthdays of the criminals, and a full dump of the control panel source code?

No.

@sednet:

In this latest incident Linode screwed up, there is no denying it. No doubt Linode are doing everything they can to fix the situation but it doesn't not distract from the point that they did screw up. Yet the fanboys deny it, they make excuses, they try to derail any criticism.

Do point out where anyone denied the issue in this thread or denied that there was a screwup. The only thing that's been posted of late is that it's ludicrous the way some of you expect linode to provide detailed instructions for how to hack their systems.

Oh boy, here we go again… X_X

@glg:

Do point out where anyone denied the issue in this thread or denied that there was a screwup. The only thing that's been posted of late is that it's ludicrous the way some of you expect linode to provide detailed instructions for how to hack their systems.

1) The post right above this one said they 'Probably' screwed up. It's not 'probably', it's a yes, they did screw up. People screw up every once in a while but Linode can make their own apologies for it. What's happening here is fanboy nonsense.

2) I don't have to point out anything to win an argument with you or anyone else. You ( or anyone else ) may base your opinions on facts, intuition, faith, or anything else you so chose. Just because you want to live in blissful ignorance of what really happened doesn't mean I want to.

3) It's entirely reasonable to ask what was compromised, how, and what has changed in order to ensure the same thing can't happen again. If you prefer the Microsoft security model of leaving the customer guessing, well you can see what kind of security history Microsoft has had.

@sednet:

What's happening here is fanboy nonsense..

I find this whole topic amusing, because both sides can be summed up as:

"what I want" –-> absolutely, no doubt, completely reasonable request

"what the other side wants" ---> fanboy nonsense

and it doesn't matter which side of the debate you're on.

@ericholtman:

@sednet:

What's happening here is fanboy nonsense..

I find this whole topic amusing, because both sides can be summed up as:

"what I want" –-> absolutely, no doubt, completely reasonable request

"what the other side wants" ---> fanboy nonsense

and it doesn't matter which side of the debate you're on.

It is not as you describe.

One side wants to know what happened to estimate future risk or simply to complain about the lack of communication. The other side has faith that whatever the problem was has now gone away, they don't care for the details.

Personally I don't run systems on faith so would like to know what happened.

@sednet:

Personally I don't run systems on faith so would like to know what happened.

What nonsense, of course you run your systems on faith.

You had faith that Linode was secure "enough" when you signed up for their VPS service.

What possible answer could they give you that would let a security breach change your "faith" level in Linode?

That's right, no answer will change the fact that they were breached, and no answer is nothing more then lip service if they promised it wouldn't happen again.

If Linode is "too risky" - move to another more secure host. Of course you'll only have faith in that new host until something happens.

If what you're hosting cannot stand up under the possibility of a security breach, VPS solutions are NOT for you.

So either accept the risks (hint: there EVERYWHERE), or get a solution that you can audit the risks yourself.

nvm, just read post above, vonskippy said it better than me

Let me toss out another distinct possibility as to why linode may have said nothing more. They may be legally unable to say anything.

If it was an inside job involving a linode employee, there could likely be an ongoing criminal investigation. If that is the case, linode may have been ordered by LE not to disclose anything at this time.

If it was instead a linode employee screwing up really badly (ie negligence, not malice like the previous case), the employee could have been fired. If that was the case, linode also would not be talking, as that can be a very dicey prospect legally as well. In my day job, I'm a manager. I know very clearly from my HR department that no matter how a previous employee left or was tossed out on his ass, I say nothing negative about it whatsoever (this probably doesn't apply to outright fraud, but luckily I haven't had to deal with that, even so, if that were the case, I'd still direct anyone asking to my HR department). Anything negative opens up a risk of a lawsuit. So, if linode publicly said "yeah, joe dipshit screwed up some security in the linode manager, so we fixed it and canned his dumb ass". Joe later tries to get another job and doesn't because of that post, Joe sues linode.

and that's just two scenarios where they'd have a lawyer saying "don't say a word publicly over what you've already said".

Gig, You missed the point there somewhat. I can accept downtime because I expect it but DNS slaves and backup mail exchangers are not built to recover from a situation where their primary is misbehaving because the machine got p0wned and is stealing my email or giving out incorrect DNS records.

Obviously a power failure or network outage will only cause downtime, not a compromise of my systems.

@glg:

nvm, just read post above, vonskippy said it better than me

That would that be the same vonskippy who was insulting people asking for ipv6 back when Linode didn't have it but other providers did.

He is so far in the pro-linode camp as to have lost all objective judgement.

@sednet:

Gig, You missed the point there somewhat. I can accept downtime because I expect it but DNS slaves and backup mail exchangers are not built to recover from a situation where their primary is misbehaving because the machine got p0wned and is stealing my email or giving out incorrect DNS records.

Obviously a power failure or network outage will only cause downtime, not a compromise of my systems.

If you don't think that anything that you can access in linode manager can't be accessed in some fashion by a sysadmin, then I just don't know what to say. You signed up for a VPS product, you're putting some faith in the owner and admins.

@glg:

If you don't think that anything that you can access in linode manager can't be accessed in some fashion by a sysadmin, then I just don't know what to say. You signed up for a VPS product, you're putting some faith in the owner and admins.

I am putting faith in them to behave legally. It would be strongly against their interests to do otherwise. Whatever happened to Linode is unlikely to be a result of linode itself behaving illegally. However as they won't give me any information whatsoever I can't be 100% sure.

I want clear information from people who know what happened. Argument and opinion from people that know no more than I do doesn't get anyone anywhere.

> They may be legally unable to say anything

If that's the case then I'd love if they told us that. Hell, if that's not the case, they might as well tell us the same thing anyway, it would get us off their backs.

Either that or just don't say that you intend to communicate openly.

@nehalem:

> They may be legally unable to say anything

If that's the case then I'd love if they told us that.

I'd accept that as a perfectly valid reason for not telling us any more. Caker didn't even say that much though.

@nehalem:

> They may be legally unable to say anything

If that's the case then I'd love if they told us that. Hell, if that's not the case, they might as well tell us the same thing anyway, it would get us off their backs.

Either that or just don't say that you intend to communicate openly.

Chicken/egg problem there. If a lawyer advised them to shut up, they lawyer isn't going to tell them to say that.

I'm not sure I've weighed in on this thread, yet. I've previously criticised linode for the IPv6 policy (indeed, I'm still not happy with it). I eventually got sufficiently annoyed with Fremont that I moved the service to Dallas.

I'm not a "fan boy" by any means.

@sednet:

One side wants to know what happened

Which is an unreasonable request. In my day job (security professional at a Fortune "small-num" company) I do have the clout to beat up vendors. If they fuck up then I do get to get details. I currently have around 50 outstanding action items with one vendor. Thursday was shouting at IBM day. However, even my company doesn't really get to shout at the likes of Microsoft, simply because we have no leverage (what we gonna do, turn off 200,000+ desktops?). (personally, yes please… :-))

However, me as an individual customer of linode… I have no such leverage. Being a security professional I note that linode have gone above and beyond the minimal requirements needed by law. They have provided a level of detail that explains the attack vector. They have not provided a "root cause analysis" (who fucked up, and how). And I don't expect one.

I'm dealing with a small company; the risks and consequences of an individual staff member screwing up are that much higher. (I know small technical service companies; I've work for them, run technology for them; my girlfriend used to work for a linode competitor. I know how they can fuck up).

And this is how you should perform your risk analysis; small companies have a risk profile that is pretty consistent. Even it caker said "we've told our staff not to drunk remote into the admin systems using open access points", what have you learned? One potential attack vector might be mitigated, but the rest remain.

Would I like to know how linode was broken into? Sure! I'd love to know! I'd love to know how Global Payments was breached, as well! (They've been less forth-coming than linode have.)

Finally I'll note that linode staff (and caker, personally) monitor or is aware of each and every post made to these forums. That they haven't responded is telling; either they can't, or they won't. If you don't like it then take your money and leave. In the "can't" case, maybe linode will be able to get some recompense for lost income; if it's a "won't" case then this is a business cost they've chosen to take.

Either way, I don't expect any more information from linode. My risk analysis takes this into account.

> or simply to complain

Ah. Well, OK then. Maybe linode needs a "flame" sub-forum.

@sweh:

Thursday was shouting at IBM day.

You have those too, eh? ;)

I am amazed at times how pervasive the "we know better than you, Mr Customer, we're IBM!" attitude can be there.

@nehalem:

> They may be legally unable to say anything

If that's the case then I'd love if they told us that. Hell, if that's not the case, they might as well tell us the same thing anyway, it would get us off their backs.

Either that or just don't say that you intend to communicate openly.
If this was they case then they probably can't reveal this information and may have been advised by lawyers to not say anything; anything they do say might be considered prejudicial to the case.

@glg:

@nehalem:

> They may be legally unable to say anything

If that's the case then I'd love if they told us that. Hell, if that's not the case, they might as well tell us the same thing anyway, it would get us off their backs.

Either that or just don't say that you intend to communicate openly.

Chicken/egg problem there. If a lawyer advised them to shut up, they lawyer isn't going to tell them to say that.

No, but they'll tell them to say that they can't comment on an ongoing criminal investigation.

@nehalem:

No, but they'll tell them to say that they can't comment on an ongoing criminal investigation.
Which would reveal that there is an ongoing investigation; something that they may have been requested (or ordered; such orders to exist) not to reveal.

@sweh:

@nehalem:

No, but they'll tell them to say that they can't comment on an ongoing criminal investigation.
Which would reveal that there is an ongoing investigation; something that they may have been requested (or ordered; such orders to exist) not to reveal.

Can they say: "trust us, we just can't tell you anymore"? :)

@nehalem:

Can they say: "trust us, we just can't tell you anymore"? :)
My mind just went through a tonne of confidence tricksters and even the snake from Disney's Jungle Book saying 'Trust me'…. :-)

Either ya do or ya don't. I doubt you're gonna get more information in the near future.

@sednet:

That would that be the same vonskippy who was insulting people asking for ipv6 back when Linode didn't have it but other providers did.
Well, since this thread has long ago jumped the shark, lets discuss IPv6.

Care to share your ginormous IPv6 traffic charts for the last 6 months?

Boy, I can't imagine how you would have survived if Linode took longer then they did to roll it out.

So post away, I can't wait to see those IPv6 traffic numbers.

As to a Linode fanboy (although as VPS hosts go, Linode was in our top 3 list when we vetted vendors) - bwahahahahahahaha - not even close, I'm a huge fan of co-location (in big shiny locked cages with video surveliance and two-factor authentication to get in), and think VPS's are toys to be played with, not host serious work (but YMMV).

@sweh:

Which is an unreasonable request. In my day job (security professional at a Fortune "small-num" company) I do have the clout to beat up vendors. If they f*** up then I do get to get details. I currently have around 50 outstanding action items with one vendor. Thursday was shouting at IBM day. However, even my company doesn't really get to shout at the likes of Microsoft, simply because we have no leverage (what we gonna do, turn off 200,000+ desktops?). (personally, yes please… :-))

However, me as an individual customer of linode… I have no such leverage. Being a security professional I note that linode have gone above and beyond the minimal requirements needed by law. They have provided a level of detail that explains the attack vector. They have not provided a "root cause analysis" (who fucked up, and how). And I don't expect one.

I'm dealing with a small company; the risks and consequences of an individual staff member screwing up are that much higher. (I know small technical service companies; I've work for them, run technology for them; my girlfriend used to work for a linode competitor. I know how they can f*** up).

And this is how you should perform your risk analysis; small companies have a risk profile that is pretty consistent. Even it caker said "we've told our staff not to drunk remote into the admin systems using open access points", what have you learned? One potential attack vector might be mitigated, but the rest remain.

Would I like to know how linode was broken into? Sure! I'd love to know! I'd love to know how Global Payments was breached, as well! (They've been less forth-coming than linode have.)

Finally I'll note that linode staff (and caker, personally) monitor or is aware of each and every post made to these forums. That they haven't responded is telling; either they can't, or they won't. If you don't like it then take your money and leave. In the "can't" case, maybe linode will be able to get some recompense for lost income; if it's a "won't" case then this is a business cost they've chosen to take.

Either way, I don't expect any more information from linode. My risk analysis takes this into account.

Probably the most reasonable and lucid post on this topic…

@sednet:

@rainkid:

I gave up on page 3 of this thread because, quite frankly, these Linode fanboys simple do not listen to reason, nor any opinion other than their own.

I've been partisipating in these forums on and off for about 8 years now, I've had various numbers of Linodes on and off during that time starting with a 80Meg UML machine. I've had very few problems and the ones I have had were resolved quickly. However I have to say the above comment is spot on. There are 'contributers' on this board that will quickly shout down any perceived criticism of linode or it's service regardless of the validity of the criticism.

IPv6 support was a good example, before IPv6 was deployed at linode and after it was deployed at many other providers anyone that asked for it on these forums was told they didn't need it and they were insulted for asking.

There were a few examples of people who canceled their accounts and didn't get a refund for the unused part. These people were soundly mocked for no good reason.

There was one chinese guy who got all frustrated at Linode wanting copies of the front and back ( PCI rules anyone? ) of his credit card and his passport. He was told he must be a scammer or somesuch thing on this forum based on nothing but his country of origin. Sad, that was a potential customer.

In this latest incident Linode screwed up, there is no denying it. No doubt Linode are doing everything they can to fix the situation but it doesn't not distract from the point that they did screw up. Yet the fanboys deny it, they make excuses, they try to derail any criticism.

These people are just a minority of forum contributers but they are vocal. I don't believe these people are sockpuppets. I believe these people just have a flawed view of reality caused by some kind of cognitive bias. I would not trust these people with a root account on any machine running anything I care about.

TL;DR - Linode screwed up, fanboys are unable to accept this so keep trying to derail this thread.

Well said Sednet.

I am hoping to go live with my business in about 10 months from now. I need to know more about this incident so that I can make a valued judgement on whether or not to continue to use Linode.

I view the incident as extremely serious and I need some kind of meaningful assurance that it won't happen again. Currently the management/owner are choosing to ignore long term (some of 8 years) customers who are asking for reasonable assurances. We are NOT asking for anything that would compromise the company.

In my experience when a company starts whistling and pretending you aren't there when asking direct questions, I get suspicious. Something smells bad and no amount of air-freshener is going to cover it up.

I just want some transparency so I can have some trust. I don't want their root admin passwords, just a reasonable business to business response instead of this stonewalling.

@tentimes:

I view the incident as extremely serious and I need some kind of meaningful assurance that it won't happen again.

You're talking about a VPS running on a host you have zero physical control over.

If anyone, anywhere from any company tells you "It won't happen again", they are lying to you. Full stop.

@ericholtman:

@tentimes:

I view the incident as extremely serious and I need some kind of meaningful assurance that it won't happen again.

You're talking about a VPS running on a host you have zero physical control over.

If anyone, anywhere from any company tells you "It won't happen again", they are lying to you. Full stop.

1. I do not care what you think. I am only interested in what Caker/Linode think.

2. I would hope they have closed down whatever loophole it was.

Also, if it was my company, I would be commissioning an independent security review with a company that had some degree of experience and respectability. Whilst we would not need to be privey to the nuts and bolts, the independent nature of the review would go some way to reassuring customers that things were under control. At the moment I am having a serious rethink on the part (notice the word PART) Linode plays in my business.

I respect Linode and up to now it has been the best in the business, but this ignoring the customer business is not on.

@tentimes:

Also, if it was my company, I would be commissioning an independent security review with a company that had some degree of experience and respectability. Whilst we would not need to be privey to the nuts and bolts, the independent nature of the review would go some way to reassuring customers that things were under control. At the moment I am having a serious rethink on the part (notice the word PART) Linode plays in my business.
I, for one, would have no confidence in an "independent security review". I assume they're incredible expensive and, well, run by people like this.

Edit: To clarify, this post is not demanding any information from Linode. I'm just commenting on the specifics of tentimes's suggestion and taking no position on the merits of the idea itself.

@mnordhoff:

I assume they're incredible expensive and, well, run by people like this.

..If that's real I'm speechless …

@mnordhoff:

I, for one, would have no confidence in an "independent security review". I assume they're incredible expensive and, well, run bypeople like this.

That is a heck of a read but it hardly means the whole security industry is a sham. There are countless companies that do this kind of thing right, just do some research before you chose one.

I'm sure no matter what the subject, or what you want to talk about, I can find a post on the interned showing how it can turn horrible and just how bad it can be but that does not mean that particular industry as a whole is bad or that your experience will be identicle.

I know I am not a security expert, I try and make everything as secure as my knowledge allows me but nobody is perfect and I don't exclusively do security, its not my specialty. That being said, if I am ever fortunate enough to make a site or an app that explodes and starts getting thousands or millions of users, well, I would certainly be remiss if I didn't get a security expert to look things over and tell me where I may be letting down my users.

Also, the story linked to was a bit different, it was a forced audit by a small payment processing company who had an idiotic employee (or set of practices).

@mnordhoff:

I, for one, would have no confidence in an "independent security review". I assume they're incredible expensive and, well, run by people like this.

Anyone who hasn't read the story linked above really should. However the complainer also acted unethically by showing a willingness to fake data.

The big mtgox.com bitcoin hack was allegedly caused by a security auditor leaking or selling the password file:

http://bitcoinweekly.com/articles/the-mtgox-attack

Security auditors are like financial auditors, the best they can give you is a 'trusted feeling' and maybe point out some weaknesses. It's a start but not a solution.

I am a lawyer and a former customer who left after the last security incident. They DO have the ability to mention if a case is forthcoming without needing to mention any specifics. Otherwise you wouldn't have media coverage of any cases.

Anyway I appreciate the opinion that it is a VPS and you should just "get what you're given" but the problem is Linode has been so fantastic all around that people set a higher benchmark than others.

I think that is why it was such a disappointment.

@sweh:

@nehalem:

No, but they'll tell them to say that they can't comment on an ongoing criminal investigation.
Which would reveal that there is an ongoing investigation; something that they may have been requested (or ordered; such orders to exist) not to reveal.

Are you saying that that is crap?

…aaaand we're at 11 pages.

WITH NOTHING NEW SAID. AT ALL.

I agree with vonskippy- never thought you'd hear that, eh?

I don't actually have an example in mind; I just wanted an excuse to say that.

@derfy:

…aaaand we're at 11 pages.

WITH NOTHING NEW SAID. AT ALL.

It would not be an 11 page thread if it wasn't for people posting redundant messages about how pointless this thread is. If you don't like it you could just ignore it instead of trying to derail valid criticism.

It's just not true that nothing has been said, Taligent made a very good point a few posts ago. It's doesn't appear to be the case that Linode legally can't say anything.

@taligent:

I am a lawyer and a former customer who left after the last security incident. They DO have the ability to mention if a case is forthcoming without needing to mention any specifics. Otherwise you wouldn't have media coverage of any cases.

You a US lawyer? Because it doesn't sound like you've heard of things like sealed investigations or gag orders from a judge. If you're advising clients to ignore those, you're not a very good lawyer. (NOTE: I doubt either of these are the case)

Also, when bringing this up, my first thought wasn't that they can't say anything, it's that their lawyers may have advised them not to say anything. If you really are a US lawyer, you'll know just how easy it is to walk into defamation suits and how easy it is to sue under employment law.

@sednet:

It's just not true that nothing has been said, Taligent made a very good point a few posts ago. It's doesn't appear to be the case that Linode legally can't say anything.

It's very obvious from your posts that you have no clue on the US legal system. Since you're not from the US, I wouldn't expect you do, but do yourself a favor and just don't post about it. It just makes you look stupid.

@glg:

@sednet:

It's just not true that nothing has been said, Taligent made a very good point a few posts ago. It's doesn't appear to be the case that Linode legally can't say anything.

It's very obvious from your posts that you have no clue on the US legal system. Since you're not from the US, I wouldn't expect you do, but do yourself a favor and just don't post about it. It just makes you look stupid.

Aww. how cute. The fanboy got all butt-hurt after my last comment.

Obviously it's not worth me responding to your butt-hurt-ness because I'm not the one who claimed to be a lawyer, go troll that guy instead.

At least we know gig isn't a Linode sock-puppet now. They would never use one as emotional and dumb as him.