The DDoS Attacks

> I'm also kind of surprised that no one is really posting here about it.

The best sources of info are IRC (https://www.linode.com/chat) and the status page (http://status.linode.com).

@piglet:

The best sources of info are IRC (https://www.linode.com/chat) and the status page (http://status.linode.com).

I've been checking the status page, and I don't know about that chat–it seems to direct to some other company's chat system.

I just wish Linode would tell us what is going on and why they are being attacked. It looks like they are implementing some kind of strategy to mitigate the attacks, so that's good. I just hope it works!

My sites in Atlanta are back online. Oops! Back down again.

I guess they are still working on the problem…..

It could be an extortion attempt (and Linode just isn't talking about the ransom demands), or someone trying to damage Linode in the marketplace. Sometimes there's just no clear reason.

I guess no one is really talking about it since there probably isn't much to say. Without knowing more, no one can make predictions about how long it will last. And due to the distributed nature of it, it's very difficult for the upstream provider to filter it.

My Newark Linode was affected a few days ago, but yeah, unfortunately it appears that Atlanta is the hardest hit. Whoever's responsible for this seems pretty dedicated, and given the different locations affected, a guess would be that the target is Linode, not a specific host/use at a datacenter.

At this point, is it really a DDoS attack? We can't even access Atlanta via other Linode data centers. So, either Linode does not have direct control over their own routers, or the problem is much worse than they claim. Otherwise, why not mask all public traffic and route internal Linode subnet traffic? Then at least we can pull our Atlanta databases and compensate/rebalance.

Please consider it. Thanks.

Pretty sure these are DDoS attacks. These things are a pain in the ass to mitigate. A popular server of mine (not hosted on Linode) was targeted for over a 9 month period. I've experienced over 80GB/second and sustained 60M pps mostly from amplification attacks, but also through other attack vectors. This went on sporadically for nearly a year and completely crippled my service regardless of all the ingress filtering, upstream filtering, proxying, and OS hardening I threw at the problem (with the help of the service provider, and their upstream providers). The attackers were asking for payment in bitcoins for the attacks to end. Ultimate, I spent too much to mitigate all the attacks (and I succeeded) but at that point, a lot of damage was already done.

I've been a Linode customer for over 10 years. Though we've had our differences in the past, I still manage about a dozen servers on Linode. I hope they're able to defend themselves against this, and hope they don't cave in to pressure. I'm sure they won't as their business is on the line.

At this time, most of my Linode servers have experienced only minimal downtime, and sporadic periods of intermittent packet loss. I'm rather impressed by how they're handling everything.

Linode already provides us with their network status updates. Any other transparency is not important at this time.

I was just curious what is really going on: i.e., is it criminals extorting for money (bitcoins), a nasty competitor, or a group who has some beef with Linode or someone they host. I'm guessing that extortion is most likely.

But if so, Linode is doing the right thing by not caving in. If Linode does cave in, it would be like feeding a bear – it will just come back later demanding more. Even if it costs Linode a lot of money to harden their infrastructure against these attacks, Linode comes out the winner because now they will be better prepared for future attacks. And, besides that, the attackers will know they can't extort Linode. Win-win.

So good for Linode, I say!

@rainkid:

Pretty sure these are DDoS attacks.

…

I'm rather impressed by how they're handling everything.
Sure, we also like Linode and give them the benefit of a doubt. However, bottom line: this was a huge disruption, and we were surprised that this type of front-door attack prevented inter-data-center connections. That just seems like poor route management to us.

Regarding the bigger picture, it's alarming that there aren't better auto-detection and auto-throttling of such volumetric attacks. We're all headed for major problems if that can't be accomplished.

@althost:

Regarding the bigger picture, it's alarming that there aren't better auto-detection and auto-throttling of such volumetric attacks. We're all headed for major problems if that can't be accomplished.

Inter-datacenter connections depend on your upstream providers. From my 9 month long attack, I switched datacenters 3 times before I was on a datacenter capable of properly routing through attacks and filtering upstream. Even then, the volumetric attacks were so varied that I had to implement other measures to drop the remaining 5% of destructive traffic that crept in. And that wasn't without it's collateral damage (including other customers in that datacenter who specializes in DDoS mitigation).

Most higher tier bandwidth providers have auto-detection/auto-throttling/auto-scrubbing in place (assuming it's paid for) - however, attacks of this size and nature need manual intervention and attention to handle. The moment you think you have the problem taken care of, the attackers change their attack a bit.

It's a total pain in the ass to handle, and during the time I was hardest hit - I would be lucky to get 2 hours sleep a night.

(My attackers did post their extortion demands on my social media accounts, which actually garnered a lot of support for me to not to give in to their demands - not that they were asking for much. This is how extortion in the digital age works - ask for a small amount, see if you cave in, then continue the attacks and extort larger amounts.)

Hi all,

This really seems to be a very big ddos attack and just a few hours ago when Atlanta problems seemed to be resolved, there is currently going on a new attack to the Linode DNS servers.

At this point i hope Linode already knows the origin of the attacker (maybe some hosting competitor or some blackhat hackers asking for money to stop the attacks, or…) and also start to think on ways to better protect their network in terms of security… i know that there is no "perfect" solution to protect from ddos and that all good solutions are very expensive.. but anyway i think Linode must have some kind of better "fight back plan".

Something like "connecting Linode network directly to Cloudflare" in this cases… just kidding.. but who know if some kind of partnership with ddos companies will do the trick..

Now talking about the clients, and we all are in that position of having our christmas holidays ruined with servers down, helpdesk tickets and many end-user clients phone calls.. i just can imagine the quantity of money lost by online stores that where down during last days.. worldwide.

At this point i hope all problems to be gone in the next few hours… but if not.. we will have ALL a major problems and the pressure of clients are making us reach to a point where we would have to move from hosting provider not by not trusting Linode, but in order to re-gain our end clients trust and having our SLA contracts to re-gain trust. Because yes,, almost all end clients don't know Linode and don't understand the hosting market.

Also does anyone know any more specific news or details about what is going on?

Luck to us all!!

@althost:

we were surprised that this type of front-door attack prevented inter-data-center connections. That just seems like poor route management to us.

Unless this has changed, in the past Linode has not had dedicated inter-DC connections. inter-DC traffic flows the same as all other traffic, over the public internet. So, it's just as susceptible to these attack vectors as your customer traffic coming in.

Is Linode going to provide a "discount" to those users who can not get access to the backend of their system? I have to make some changes to the nameservers and set up things for some clients but I can not do this because I can not access linode.com. Also how safe is our credit card information? I have all ready been the victim of some of the biggest breaches to date and this is just getting ridiculous.